jump to navigation

So, what about (Perl 6) dependencies? 2017-05-28

Posted by claudio in Uncategorized.
Tags: , , , ,
trackback

DependenciesWhen I need to program something, most of the time I use Perl 5, Go or Perl 6. Which one depends on the existence and maturity of libraries and the deployment strategy (and I must admit, probably my mood). Most applications I write at work are not that big, but they need to be stable and secure. Some end up in production as an extension or addition to the software that is the core of our authentication and authorisation infrastructure. Some programs are managed by other teams, e.g. of sysadmin-type applications like the monitoring of a complex chain of microservices. Finally, proof of concept code is often needed when designing a new architecture. I believe that If your software is cumbersome/fragile to install and maintain, people won’t use it. Or worse, stick with an old version.

Hence, I like CPAN. A lot. I love how easy it is to download, build, test and install Perl 5 libraries. cpanminus made the process even more zero-conf. When not using Docker, tools like App::Fatpacker and Carton can bundle libraries and you end with a mostly self-contained application (thx mst and miyagawa).

Go, a compiled language, took a different path and opted for static binaries. The included (Go) libraries are not downloaded and built when you’re deploying, but when you’re developing. Although this is a huge advantage, I am not too fond of the dependency system: you mostly end up downloading random versions from the master branch of random Github repos (the workaround is using external webservices like gopkg.in, not ideal). The only sane way is to “vendor in” (also with a tool) your dependencies pretty much the same way Carton does: you copy the libs in your repository (but still without versioning). I hear the Go community is working at this, but so far there are only workarounds.

In the Perl 6 world, the zef module manager does provide a kind of cpanminus-like experience. However, in contrast with cpanminus it does this by downloading code from a zillion Github repo’s where the versioning is questionable. The is no clear link between the version fixed in the metadata (META6.json) and branches/tags on the repo. Like mentioned above, Go gets away with this due to static compiling, although the price is high: your projects will have dozens of versions of the same lib, probably even with a different API… and no way to declare the version in the code.

The centralised Perl 5 approach approach is fairly complex. It works because of the maturity of the ecosystem (the “big” modules are pretty stable) and the taken-for-granted testing culture (thank you toolchain and testing people!). Actually, in my opinion, the only projects that really solved the dependency management problem are the Linux & BSD distributions by boxing progress in long release cycles. Developers want the last shiny lib, so that won’t work.

The Perl 6 devs have no static compiling on the agenda, so it’s clear that the random Github repo situation is far from ideal. That’s why I was pretty excited to read on #perl6 that Perl 6 code can now be uploaded to CPAN (with Shoichi Kaji’s mi6) and installed (with Nick Logan’s zef). Today, the Perl 6 ecosystem has neither the maturity of the one of Perl 5 nor its testing culture. The dependency chain can be pretty fragile at times. Working within a central repository with an extensive and mature testing infrastructure will certainly help over time. One place to look for libraries, rate them, find the documentation, and so on. Look at their state and the state of its dependencies. This is huge. But I don’t think that CPAN will fix the problems of a young ecosystem right away. I think there is an opportunity here to build on the shoulders on CPAN, while keeping the advantages of a new language: find out what works.

Personally, I would love to have a built-in packaging solution like Carton –or even App::Fatpacker– out of the box. I think there is something to be said for the “vendoring-in” of the dependencies in the application repo (putting all the dependencies in a directory in the repo). The Perl 6 language/compiler has advantages over Perl 5. You can specify the compiler you target so your code doesn’t break when your compiler (also) targets a more recent milestone (allowing the core devs to advance by breaking stuff). Soon, you’ll be able to You can even load different versions of the same library in the same program (thx for the correction, nine).

The same tool could even vendor (and update) different version of the same library. At “package time” this tool would look at your sources and include only the versions of the libraries that are referenced. The result could be a tar or a directory with everything needed to run your application. As a bonus point, it would be nice to still support random repos for libraries-in-progress or for authors that opted not to use CPAN.

My use case is not everyone’s, so I wonder what people would like to see based on their experience or expectations. I think that now, with the possibility of a CPAN migration, it a good time to think about this. Let’s gather ideas. The implementation is for later :).

Advertisements

Comments»

1. ugexe - 2017-05-28

We have p6c testers. A Perl6 test report was the first report sent to the new v3 CPAN testers API. We already support cpan and the typical repo-style ecosystem as complimentary (not exclusive). You can practically vendor in solutions with just zef (using install-to), and you can in entirety in combination with CompUnit::Repository::Mask. We can already load multiple versions of the same module in different scopes.

This just leaves static compilation and the ecosystem maturity. Not much we can do about maturity (it takes time), but the lead architect of moarvm is accepting funding if anyone is interested in static compilation as a priority.

claudio - 2017-05-28

Hi,

I don’t think static compilation is a quick win. Vendoring is. So I am not looking in that direction to make Perl 6 easily deployable. Core devs have other priorities and I agree with them.

It’s possible that zef (thank you!) will be the tool to do that, but I don’t think it’s at the moment. Don’t get me wrong, I am a fan.

I think that versioning would be somewhat troublesome at the moment. You would need to “rm -rf ~/.perl6/*” every time you use “-to”. I think this is a feature of zef (keeping state of already installed modules), just working on an other use case.

C.

claudio - 2017-05-28

ugexe,

And by the way, something like your https://github.com/ugexe/Perl6-CompUnit–Repository–Tar may be a logical step if a vendoring dir is in place…

2. nine - 2017-05-29

If you want to use precise versions of dependencies in your code, you just have to do exactly that: use Foo:ver(v1.23);
Add the info to your META6.json in the same way: ‘depends’: [‘Foo:ver(v1.2.3)’] and zef or other installers will ensure that version is actually installed. There is no need to put those modules into separate repositories, as you can install multiple versions of the same module (and indeed, use them in the same program in different scopes).

There’s still the ecosystem issue with multiple git commits sharing the same version number, but that will now hopefully disappear with the move to CPAN. Indeed, having people install the latest release rather than a development version was the primary motivation for me to upload Inline::Perl5 to CPAN.

3. Wouter Verhelst - 2017-05-29

From a distribution’s point of view, having multiple (possibly incompatible) versions of the same library loaded is a nightmare. That’s true from an operator’s point of view, too, fwiw.

I blogged about this a while back: https://grep.be/blog/en/computer/cluebat/rails_followup/

claudio - 2017-05-29

I agree. Distributions solve the problem on their own way. I expect a well tested-together collection of modules from the distribution. For a lot of (packaged) software is this a perfect fit (e.g. I run a big Radiator (RADIUS) installation using the Perl OS packages).

On the other hand, everything extra is the responsibility of the developer/admin. Vendoring is a lot saner than globally installed modules like CPAN/Rails/etc do by default.

claudio - 2017-05-29

I wasn’t aware the versioning was ready in place (I haven’t seen it in the wild). Great to hear! I’ll adapt the text. If the ecosystem moves to CPAN, we’ll indeed have a solution for the upstream versioning problem because we’ll have real releases instead of a master HEAD that varies over time (so installing older releases afterwards is not straight forward).

The way I imagine the dependency chain is a little different than your practice, although what you say is a logical consequence of what I wrote. I was thinking on a more relaxed versioning for libraries (just following CPAN practice, e.g. min version and max when needed). The version-pinning is done by the application using the libraries. So it’s more of a deployment strategy than one focused on the API/ABI: I want to take the self-contained lot, put it on a remote server or container and run it.

C.

4. 2017.22 Up Handle Encoding | Weekly changes in and around Perl 6 - 2017-05-29

[…] So, what about (Perl 6) dependencies? by Claudio Ramirez. […]

5. SmokeMachine - 2017-05-30

Please, take a look at 6pm: github.com/FCO/6pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: