100 Days Of Open Source (2017)
A commit a day keeps boredom away
Inspired by the 100DayProject — and insufficiently-well prepared for 100DaysOfIndieWeb —, I commit myself to a 100 Days of Open Source challenge, starting January 1st, 2017. For more than 3 months I will contribute at least one meaningful commit a day to a public Open Source project or a similar community effort.
There's a lot of Open Source software involved in my daily work and not at least I'm earning a major part of my income with using and implementing the work of others who have dedicated their efforts to the public. In turn, I've created a couple of Open Source projects myself which steadily need support, maintenance and improvement.
When Jeremy wrote about his 100 Days project back in 2015, I got fascinated by the idea of challenging myself in some way, just to see if I could make it. In November 2015 I incidentially started a Github streak that went on for two or three weeks before I noticed that it could probably keep on going for a while as I had lots of public projects going on at that time. It soon became a kind of silent game with myself that finally lasted until August 2016 — for almost 300 days. I never told anyone about it and quit as silently as I had taken it up. Contrary to all doubts you might have, I had a lot of fun doing this!
Officially and publicly accepting the 100 Days challenge basically brings two differences: I'm telling you about it, and I will document my progress. Apart from that, I'm pretty sure I'll have big fun again and hope to get a lot of stuff done during these 100 days. Bring it on!
Day 1: micrometa with experimental JSON-LD support (2017-01-01)
In reaction to a user's request for JSON-LD support in micrometa, I decided to start the challenge right here and published a first release with experimental JSON-LD parsing capabilities.
Day 2: Improved micrometa demo website (2017-01-02)
JSON-LD support had also to be added to the micrometa demo website and while being at it, I decided to improve its overall look and user experience a bit.
The sections of micro information items are now collapsible, plus the item type (Microformat 1+2, HTML Microdata or JSON-LD) is explicitly shown by an icon. Wherever possible, URLs are directly linked so you can test the target resources.
Just give it a try if you've got a site that you want to examine for embedded micro information.
Day 3: CoderDojo website & micrometa bugfixes (2017-01-03)
There are several smaller things I could achieve today:
Day 4: Brushed-up squeezr release (2017-01-04)
After pretty exactly 3 years I spent some time on squeezr and pushed out a slightly improved release today. In particular, I closed some long-standing issues, refactored and cleaned up the codebase and moved everything over to a more Composer like installation. I have no idea if anyone is still using squeezr in production, but I'm still amazed by it's simplicity and efficiency.
Day 5: Released new TYPO3 extension & bugfixes (2017-01-05)
Today I had to focus on client projects so I couldn't update our TYPO3 squeezr extension as I originally planned to. However, there were still some of our Open Source projects involved so I got plenty of chance to do some bugfixes and improvements:
- I published our TYPO3 XML sitemap extension for the first time. It's undocumented at the moment but I plan to add some docs shortly.
- I fixed some minor bugs in our Yeoman Generator which was necessary to setup a new client project.
- I enabled TYPO3 8 support in my still-under-development TYPO3 Fractal bridge so that we can experiment with a component library in our new project that will be kicked-off next week.
Day 6: squeezr TYPO3 extension (2017-01-06)
While it was a public holiday here today, I started refactoring our squeezr TYPO3 extension, not least because this very website actually depends on it but I had to disable squeezr because something stopped working. I also wanted to transform it into composer mode so that we can use the extension in our future projects. However, it turned out that I needed to change the whole layout for that, so I will probably drop support for non-composer mode installations with the next release. I hope to finish the refactoring tomorrow, but I'm at least half way through.
Day 7: squeezr TYPO3 extension Pt. 2 (2017-01-07)
I finished the refactoring of our squeezr TYPO3 extension and published the new release 1.5.0. The extension is now ready for use with TYPO3 7+ in composer mode while the installation / configuration is even more frictionless as before. As we will only use composer mode for TYPO3 in the future, I decided to not make the extension compatible with the old installation style anymore. So this very website will not see squeezr support coming back until its end of life.
In addition to the extension itself, our Yeoman Generator also had to be updated in order to support the new release. And while being on it I also fixed some minor bugs over there ...
Day 8: Visualizing composer dependencies (2017-01-08)
Yesterday I stumbled about the graph-composer package that can be used to visualize the Composer dependencies of a PHP project. It does exactly what I was looking for — in preparation of a new Yeoman generator for apparat style PHP projects I was looking for ways to analyze package dependencies in a visual way.
Unfortunately, the latest graph-composer release is only able to visualize all dependencies of a project while I'd really like to skip the development dependencies for my purposes. It turned out that someone had already committed a pull request to introduce a
--no-dev option — which hadn't been merged for months though. I also found that the PR didn't really work as I would've expected: While it does suppress the immediate dev dependencies of the root package, it still lists those packages (as additional root nodes) that have non-development dependencies themselves (which makes it quite useless I think).
Long story short: I quickly rolled my own PR proposing a
--no-dev and a
--dev-only option (the way I would expect them to work). The PR isn't merged yet, but you can use my fork to see the additions in action — it works like a charm. As a result, I already updated all apparat packages with a dependency graph in their README. I really like it!
Day 9: Sketching a PHP project Yeoman generator (2017-01-09)
This was the first day of the year when our whole team (and most of our clients) got back to work, so we spent it mostly in a team event and extensive kick-off meeting. When I finally found some spare time late at night I was just too exhausted to do high-key programming. So what I did is analyze some of my latest PHP projects and sketch the functionality of a Yeoman generator for PHP projects I want to push out next.
I plan to use a main generator that is run whenever a project gets initially started. Additionally, there will be a couple of sub generators that could also be run at a later stage to add functionality like dokumentation, code analysis etc.
Days 10-12: PHP project Yeoman Generator (2017-01-10..12)
I didn't get to document my Open Source activities lately, which pretty much reflects my current workload, but this doesn't mean that I was lazy. Let's wrap it up:
Most importantly, I started implementing the Yeoman Generator for PHP projects I was mentioning earlier. During the last year I experimented a lot with different approaches for structuring PHP projects. I became a huge proponent of Composer and started splitting my apparat code into separate libraries — which automatically raised the question of how they relate to and reference each other. I made several attempts to give them a consistent structure and explored different styles of software architecture. I finally settled with a custom interpretation of The Clean Architecture, mixed with a lot of opinion I guess. I can, by the way, totally recommend reading The Clean Architecture in PHP and Domain-Driven Design in PHP — both helped me a lot to sort these things out.
As my setup has proven quite useful in several projects, I think I should document my findings in some way. Also, I'd like to ease the process of scaffolding new PHP projects as this can be a quite tedious task. So casting my architecture into a Yeoman Generator seems to make a lot of sense — even others could benefit this way. On another note, I'm recently having a lot of fun doing stuff like this as it gives me the chance to put my newly acquired ES6 knowledge into practice. ;)
Yeoman got a major new release recently and I still have to get used to the things that have changed since the last stable version. I started implementing the main generator that sets up the project foundation and will be accompanied by several subgenerators which add additional functionality like a git repository, CI and code analysis tools as well as documentation features. In particular, I want the subgenerators to be usable independently so that you can add these features at any later stage as well. I'll continue working on the generator during the next days.
Besides working on the Yeoman generator,
Day 13: TYPO3 component library & living styleguide with Fractal (2017-01-13)
Some weeks ago, I started to implement an experimental bridge between TYPO3 and Fractal, Mark Perkins' Node.js based component library and styleguide tool. There are several reasons why we'd like to have living styleguides alongside our future TYPO3 projects:
- Being able to view and check the components of a project in an isolated way and outside the context of the whole page would help us to avoid unwanted side effects and interferences.
- Having a clean known target state of each component would enable us to do automatic visual testing against the production website — which would be a huge progress.
Coupling TYPO3 with Fractal isn't trivial though. It's my most important requirement that we don't have to clone our templates and code for the purpose of creating a styleguide but use true live assets at all levels. I'm sure the former would inevitably lead to outdated styleguides very soon and also introduce an increased risk of errors. So what I did first was create a TYPO3 extension that does two things:
- It exposes "components" via a command line API in JSON format (see below for what can make a component and how you define that). In theory, any other tool similar to Fractal could use this component data as well.
- It provides a command line API for rendering any of the exposed components. This way, Fractal can use TYPO3 as a "rendering / templating engine" for the components.
To begin with, I distinguished three component types that reflect different ways of how TYPO3 creates output:
- A component could be rendered by TYPO3's MVC framework "extbase", which involves calling a controller action and rendering a Fluid template (which is TYPO3's templating engine).
- A somewhat simpler component would be created by simply rendering a standalone
- Finally, components could be created by TypoScript, which is TYPO3's configuration syntax.
The bridge provides an abstract PHP class for each of these component types that must be extended in order to expose a particular component to the command line API. The navigation structure of your components in Fractal is derived from the directory structure you use for exposing the TYPO3 components.
The second part of the TYPO3-Fractal bridge is a small Node.js module that extends Fractal and adds a custom CLI command. Running the command extracts the components from TYPO3 and creates the files and directories needed to show the components in Fractal's web view.
Fractal defaults to rather simple templating engines like Handlebars, Nunjucks, etc., which typically use static templates and some input parameters to render their output. TYPO3's rendering, however, depends on a huge configuration array and may involve complex database queries, GET request data and so on, so I don't see a realistic chance to delegate the rendering of the components to something other than TYPO3 itself. Therefore, the second purpose of the Node.js module is to provide a rendering adapter to use TYPO3 as a templating engine.
To showcase the component library in realtime it is necessary to use Fractal's builtin development server. As I don't want to fire it up manually each time and keep a console window open as long as it is running, I created an OpenRC init script for our Gentoo boxes that runs the Fractal development server as a system service. Finally, I set up an Apache reverse proxy which gives us a nice domain for the library, et voilà — as of today, our first simple proof-of-concept TYPO3-Fractal-bridge living component library is up and running! It's still way too early to give you some detailed screenshots (especially as we now have to start fleshing out some components you could view), but I'll make up for this as soon as possible.
Day 14: IndieWebCamp prospectus (2017-01-14)
As in 2016, I'm going to organize an IndieWebCamp in Nuremberg again this year. It's going to take place from May 20th to 21st and will be a headliner event of the Nuremberg Web Week once more. Last year's camp — with more than 40 attendees — was really well received and I'm very much looking forward to welcoming lots of nice people from all over Europe and the world to my hometown again. Tickets are completely free, the registration is open since July 2016, so go ahead an grab your's in case you plan to join us.
In fact, the 2017 camp might be even more exciting than last year as there will be another IndieWebCamp in Düsseldorf only one week earlier. So Marc and I are thinking about having a little extra schedule for those who plan to attend both camps and making it a first ever "IndieWebWeek". I don't have any details yet, but please stay tuned for further updates (and get in touch with me if you have interesting project ideas for such a week).
I would also like to establish a "Travel Assistance Fund" this time and help people from underrepresented or underprivileged groups to join us and attend the camp by helping them with their travel and lodging expenses. For this purpose, I prepared a little sponsoring prospectus (sorry, German only) that I can use to approach potential sponsors. I'm extremely happy and proud that SUSE Linux, main sponsor of last year's camp, already committed for the same (Platinum) package again. Yay!
Day 15: CoderDojo Nürnberg #5 (2017-01-15)
On Sunday we ran our 5th CoderDojo Nürnberg, and for several reasons this little anniversary was truly special. It was the first of 9 dojos we will run in 2017 and was booked for 2 months. We were almost 60 people — more than ever — and had quite precisely a 50% ratio of boys and girls (with more than 30 children in total). It was the first dojo after our successful crowdfunding campaign back in November so we had the pleasure of inaugurating our brand new 50 chairs! Again a big shout-out to all the supporters we had — you guys really rock!
As always, the dojo was exhausting, but really rewarding. There's so much we can learn from each other (and especially from the kids). As the demand for our dojos is constantly rising we decided to plan all the 2017 dates in advance and slightly increase the frequency while reducing the participants from 50 to 40. But still — and what could be better? — our next two dojos in February and April are already booked out!! I'm admittedly super proud of that. If you're interested you can find the full set of photos on Flickr.
Day 16: NUEWW organization & Microformats podcast (2017-01-16)
Monday was a mixed bag. I didn't have a lot of time for working my personal projects but spent a lot of time in scrum meetings and an internal mini-workshop with my co-workers. On the afternoon, I brought Umi to a Karate test training — and believe it or not, I'm truly considering starting Karate myself again as well!
After that, I had a short cameo appearance at the first Nürnberg Web Week organizer meeting in 2017 before heading back to the office to record a Workingdraft revision with Hans and Anselm. They had invited me to talk a little bit about micro information in websites and so I tried my best to give them a summary and comparison of some formats available to developers, namely Microformats, HTML Microdata / schema.org, JSON-LD and a little bit of RDFa. As some of you know I can get really passionate about these sort of things, but I also tend to start shaving the yak when talking about them, so I really hope I managed to hit the sweet spot. In any case, I had a lot of fun talking with you guys — thanks again! I'll link the episode here as soon as it got published.
Finally, I managed to spend some time further improving my PHP project Yeoman generator. It now nicely integrates with the first subgenerator for initializing a git repository.
Days 17-20: Yeoman composability & improving the TYPO3-Fractal-Bridge (2017-01-17..20)
The CoderDojo is hardly over — and of course immediately the flu kicks in, seriously hampering my productivity. Unfortunately, it's not possible to take a few days off at the moment, otherwise I'd probably do that.
At the office, our first project using our TYPO3-Fractal-Bridge is in full swing and runs quite smoothly so far. Of course there's always something to add or improve and we still need to get familiar with this new way of organizing our code, but by and large we are really pleased with the results so far.
While I commited a couple of minor additions to the bridge during the last few days, there's one particular issue I still have to solve (and any hint in this case is highly appreciated): While it's not a problem to render e.g. a
TEXT TypoScript component to Fractal, an
HMENU won't do at the moment. To be more precise, the
stdWrap of that very
HMENU will get rendered, but not so its
TMENU children. I guess it must have something to do with the the bridge using TYPO3's CLI while the
HMENU needs a "real" web server context (for URL generation or similar). Of course I'm already simulating a Frontend environment when calling the bridge, but I must have missed something small but important ...
When I didn't feel too exhausted in the evenings I continued working on the PHP project Yeoman generator and added all the subgenerators I need. I learnt a lot about Yeoman and composability — especially that it seems to be impossible to successively modify one and the same file in multiple subgenerators that have been nested using
composeWith(). Only the latest modification will land on disk in these cases. I suppose this is because of the in-memory file system Yeoman uses. The documentation states that files are written to disk only once when the generator finishes. I guess this is the reason why it's impossible for intermediate subgenerators to pick up and modify the (already altered) state of an existing file.
There exist certain dependencies between my (sub)generators, but I'd like to be able to use any entrypoint for an installation. So if I directly call
yo php:codeclimate, the
codeclimate subgenerator should be capable of autonomously pulling in the
git and, as a consequence, the
main generator before running himself. In order to refine particular resources from within multiple subgenerators (e.g.
composer.json or the Travis CI configuration), I'll probably have to roll my own simple intermediate file system that's shared among the subgenerators. Also, I'll have to refactor the way my generators nest each other. Apart from these "organizational" tasks, the subgenerators themselves are almost feature complete now.
Day 21: Certbot & HTTP Public Key Pinning (2017-01-21)
Today is Umi's 6th birtday and although I'm still feeling quite sick I managed to process a huge pile of (digital) "paperwork" that was long overdue. I must've read, answered and archived at least 50 emails. The one mentionable thing is, however, that I finally managed to document my experiences with enabling HTTP Public Key Pinning (HPKP) with Certbot on our Gentoo boxes. Did you try that yourself? Is there anything I could improve?
Day 22-25: Progressing (2017-01-22..25)
The last couple of days have been very busy but rather unspectacular in terms of presentability. Our first TYPO3 / Fractal based component library is steadily growing and we're really enjoying the new tool a lot! Compared to developing and previewing our components as part of the real website, Fractal forces us to focus a lot more on the clean structure, styling and behaviour of each single component. Both the process and the result feels a lot cleaner and more robust than ever before. I managed to solve the menu rendering problem mentioned earlier.
At today's Homebrew Website Club I finished refactoring my PHP project Yeoman generator — it now uses a custom approach to combine the different subgenerators so that they can share their output with each other. I created a test installation and repository to test the various services that connect to the project (i.e. Travis CI, Coveralls, Code Climate, Scrutinizer and Read The Docs). Apart from Read The Docs they all seem to run smoothly! I collected a couple of smaller improvements which I will work on during the next couple of days, but by and large the core functionality should be finished now.
Day 26-28: Release the generator! (2017-01-26..28)
Today I finally released a first version of my PHP project Yeoman generator to the npm registry after giving it the final touches and a proper documentation over the last 3 days. I still need to document the Clean Architecture principles I try to adhere to in my PHP projects, but that's more of a writing job which has to follow next. If you fancy trying out the generator, please let me know what you think!
Days 29-33: Jotting down an architecture (2017-01-29..02-02)
While the last days have been very much driven by dates and obligations, I'm now looking forward to spending a couple of days in the mountains of South Tyrol as of tomorrow — yay! Hopefully this will give me the chance to write a little bit more regularly about the progress I make with my Open Source projects.
At the office, the whole team is working full steam on the project using the new TYPO3-Fractal-Bridge, and we're still making lots of positive experience every day. So you can probably imagine my excitement when I received this question today:
Hello @jkphl, we are looking for a styleguide and working with Typo, we'd be very interested to know if you pursued in this direction.— Sven | WEBDESIGN (@scosnuau) February 2, 2017
It's still too early to summarize our experience and give you an insight into the component library we're about building, but I'll probably improve the bridge's documentation during the next couple of days so that Sven can try it out himself. Curious what his opinion might be!
On another note, I started to formulate and jot down the architectural principles I used for my latest projects, mainly the set of apparat libraries. As simplicity is one of the main goals of my approach, it is important for me to put things as simple and clear as possible. Also, I'm re-reading a lot of material and cross-checking my decisions with traditional techniques. If you know me, it won't surprise you that I'm an incredibly slow and pedantic writer — which to improve is, by the way, one of the reasons for this seemingly endless writing exercise ;). It will probably take me another couple of evenings to finish a (then hopefully still compact) wrap-up of the architectural style I used.
Lastly, I met with some fellow CoderDojo mentors today to prepare our next dojo on February 19th. We talked about some event requests I got, planned the arrangements we have to make for the kids and decided to spin-off a dedicated Github organization to improve the development of the SushiCards we're creating. It's always both productive and entertaining to meet the other mentors and I'm really grateful to know all these great people!
Days 34-35: Refactoring in Tyrol (2017-02-03..04)
Yesterday at the office I tried to tick as many boxes as possible before we left for South Tyrol. It turned out that the TYPO3-Fractal-bridge had a major problem with non-core content element types (
fluidcontent records in this case). I traced the problem back to the fact that we used the TYPO3 CLI to render the components for Fractal, which meant that we always had to simulate a frontend environment for things to work properly. Well, not too properly as it turned out...
We arrived here yesterday late in the evening and guess what — of course my friend, the little flu, had joined us as a blind passenger and said hello this morning :/. So I took it easy today, stayed at home most of the time and took the chance to fundamentally refactor the component rendering. The bridge now uses a "real" frontend plugin to request the rendered components from TYPO3, which solves the content element problem on the one hand but breaks the Extbase controller style components on the other. However, we don't necessarily need this component type at the moment so I'll fix this when I'm back home in Nuremberg.
Day 36-41: A Clear Architecture (2017-02-05..10)
After several nights of reading, writing and re-rephrasing I finally published the first version of The Clear Architecture, my personal implementation of Robert C. Martin's Clean Architecture. This is the structure I used for my apparat libraries after several unsatisfactory attempts to find the perfect architecture, and it's also the layout I'm promoting with my PHP project Yeoman Generator.
Besides apparat, I also applied this approach to several other projects and it seems to work pretty well. I'm sure there's a lot of potential for improvement but in any case I'm proud of having taken the first step by writing it all down. Should you decide to give it a shot and start a project based on the Clear Architecture, please let me know — I'm super excited! My next step will be to revise my apparat libraries and check if I really sticked to all the rules I documented (*cough*).
Day 42: Back to the party (2017-02-11)
I'm back from my short holidays (which were awesome by the way! If you ever get the chance, visit Corvara in winter time!) and as you can imagine there are quite some things I have to catch up to. Open-source-wise I started with some house cleaning:
- I added the missing support for JSON-LD value arrays to the micrometa parser.
- I renamed my PHP project Yeoman Generator to "clearphp" in favor of yesterday's Clear Architecture post.
Also, I took the following tweet as an opportunity to use the generator and start a new little project: There will soon be a simple and lightweight RDFa Lite parser written in PHP (which will then become a part of micrometa, possibly also replacing the HTML Microdata parser).
Days 43-48: RDFa Lite 1.1 & HTML Microdata parser (2017-02-12..17)
During the last couple of days, I started working on a brand new RDFa Lite 1.1 and HTML Microdata parser library in PHP and I'm happy to announce that there's a first working version available on Github as of now. The Microdata implementation doesn't quite cover the W3C Working Group Note yet and the whole package is still completely undocumented, but I'll continue working on this during the next couple of days. As far as I can tell, there's no similar RDFa Lite extraction library on Packagist yet, so there might be some demand for it?! Also, I used the Clear Architecture to layout the library and will try to keep it a kind of reference implementation. Stay tuned for further updates.
Days 49-51: CoderDojo & Micro information parser release (2017-02-18..20)
Yesterday we ran our 6th CoderDojo here in Nuremberg — and unsurprisingly we had a blast again! In January we decided to slightly reduce the number of attendees down to 40 and increase the frequency instead. This time a couple of ninjas and mentors dropped out because of colds, so compared to the last dojos it was rather quiet (but not at all less productive). The kids hacked away like crazy, two of them launched their first-ever hand-coded homepages, we've seen an incredibly well done Scratch game and even the smallest ones shot a couple of truly creative stop motion clips. (Just in case you wonder: the kids pick the topics and titles of their clips themselves. No idea why most of them are rather martial!?)
On another note, I'm happy to announce the very first release of my new RDFa Lite 1.1 / HTML Microdata parser library. It's written in PHP and quite incidentially a kind of reference implementation of the Clear Architecture I proposed recently. As far as I can tell, it's the only package available on Packagist that's dedicatedly consuming and extracting RDFa Lite 1.1 out of web documents. The fact that RDFa Lite and HTML Microdata are quite similar in terms of implementation has encouraged me to create this hybrid parser. My next goal will be to use the new parser and bring RDFa Lite support to micrometa.
Finally I realize that this is day 51 of my 100 Days of Open Source and I made it half way through already — yay! \o/