Will >> Will's blog
Sun, 20 Jul 2014
This is the status report for development on Input. I publish a status report to the input-dev mailing list every couple of weeks or so covering what was accomplished and by whom and also what I'm focusing on over the next couple of weeks. I sometimes ruminate on some of my concerns. I think one time I told a joke.
Last status report was at the end of June. This status report covers the last few things we landed in 2014q2 as well as everything we've done so far in 2014q3.
Landed and deployed:
- 6ecd0ce [bug 1027108] Change default doc theme to mozilla sphinx (Anna Philips)
- 070f992 [bug 1030526] Add cors; add api feedback get view
- f6f5bc9 [bug 1030526] Explicitly declare publicly-visible fields
- c243b5d [bug 1027280] Add GengoHumanTranslater.translate; cleanup
- 3c9cdd1 [bug 1027280] Add human tests; overhaul Gengo tests
- ff39543 [bug 1027280] Add support for the Gengo sandbox
- 258c0b5 [bug 1027280] Add test for get_balance
- 44dd8e5 [bug 1027280] Implement Gengo Human push_translations
- 35ae6ec [bug 1027280] Clean up API code
- a7bf90a [bug 1027280] Finish pull_translations and tests
- c9db147 [bug 1027286] Gengo translation system status
- f975f3f [bug 1027291] Implement spot Gengo human translation
- f864b6b [bug 1027295] Add translation_sync cron job
- c58fd44 [bug 1032226] en-GB should copyover, too
- 7480f87 [bug 1032226] Tweak the code to be more defensive
- 7ac1114 [bug 1032571] CSRF exempt the API
- ac856eb [bug 1032571] Fix tests to catch csrf issues in the api
- 74e8e09 [bug 1032967] Handle unsupported language pairs
- 74a409e [bug 1026503] First pass at vagrantification
- a7a440f Continued working on docs; ditched hacking howto
- 44e702b [bug 1018727] Backfill translations
- 69f9b5b Fix date_end issue
- e59d4f6 [bug 1033852] Better handle unsupported src languages
- cc3c4d7 Add list of unsupported languages to admin
- 32e7434 [bug 1014874] Fix translate ux
- 672abba [bug 1038774] Hide responses from hidden products
- e23eca5 Fix a goof in the last commit
- 6f78e2e [bug 947767] Nix authentication for API stuff
- a9f2179 Fix response view re: non-existent products
- e4c7c6c [Bug 1030905] fjord feedback api tests for dates (Ian Kronquist)
- 0d8e024 [bug 935731] Add FactoryBoy
- 646156f Minor fixes to the existing API docs
- f69b58b [bug 1033419] Heartbeat backend prototype
- f557433 [bug 1033419] Add docs for heartbeat posting
Landed, but not deployed:
- 7c7009b [bug 935731] Switch all tests to use FactoryBoy
- 2351fb5 Generate locales so ubuntu will quite whining (Ian Kronquist)
Current head: 7ea9fc3
At a high level, this is:
- Landed automated Gengo human translation and a bunch of minor fixes to make it work more smoothly.
- Reworked how we build development environments to use vagrant. This radically simplifies the instructions and should make it a lot easier for contributors to build a development environment. This in turn should lead to more people working on Input.
- Fixed a bug where products marked as "hidden" were still showing up in the dashboard.
- Implemented a GET API for Input responses. (https://wiki.mozilla.org/Firefox/Input/Dashboards_for_Everyone)
- Implemented the backend for the Heartbeat prototype. (https://wiki.mozilla.org/Firefox/Input/Heartbeat)
- Also, I'm fleshing out the Input section in the wiki complete with project plans. (https://wiki.mozilla.org/Firefox/Input)
Over the next two weeks
- Continue fleshing out project plans for in-progress projects on the wiki.
- Gradient sentiment and product picker work.
What I need help with
- We have a new system for setting up development environments. I've tested it on Linux. Ian has, too (pretty sure he's using Linux). We could use some help testing it on Windows and Mac OSX.
Do the instructions work on Windows? Do the instructions work on Mac OSX? Are there important things the instructions don't cover? Is there anything confusing?
- I'm changing the way I'm managing Fjord development. All project plans will be codified in the wiki. A rough roadmap of which projects are on the drawing board, in-progress, completed, etc is also on the wiki. I threw together a structure for all of this that I think is good, but it could use some review.
Do these project plans provide useful information? Are there important questions that need answering that the plans do not answer?
If you're interested in helping, let me know! We hang out on #input on irc.mozilla.org and there's the input-dev mailing list.
I think that covers it!
Tue, 01 Jul 2014
I'm going to start doing quarterly post-mortems for Input development. The goal is to be more communicative about what happened, why, what's in the works and what I need more help with.
NB: "Fjord" is the name of the codebase that runs Input.
Bug and git stats
Bugzilla ======== Bugs created: 63 Bugs fixed: 54 git === Total commits: 151 Will Kahn-Greene : 142 (+14758, -4599, files 438) ossreleasefeed : 3 (+197, -42, files 9) Anna Philips : 2 (+734, -6, files 24) Joshua Smith : 2 (+65, -31, files 5) Swarnava Sengupta : 1 (+2, -2, files 1) Ricky Rosario : 1 (+0, -0, files 0) Total lines added: 15756 Total lines deleted: 4680 Total files changed: 477
We added a lot of lines of code this quarter:
- April 1st, 2014: 15195 total, 6953 Python
- July 1st, 2014: 20456 total, 9247 Python
That's a pretty big jump in LOC. I think a bunch of that is the translation-related changes.
5 non-core people contributed to Fjord development.
I spent some time over the weekend finishing up Vagrant provisioning script and rewriting the docs. I'm planning to spend some more time in 2014q3 reducing the complexity and barriers for setting up a Fjord development environment to the point where someone can contribute.
Additionally, I'm planning to create more bugs that are contributor-friendly. I started doing that in the last week. I think a good goal for Input is to have around 20 contributor-y bugs hanging around at any given time.
Site health dashboard: I wrote a mediocre site health dashboard that's good enough to give me a feel for how the site is performing before and after a deployment. This still needs some work, but I'll schedule that for a rainy day.
Client side smoke tests: I wrote smoke tests for the client side. I based it on the defunct input-tests code that QA was maintaining up until we rewrote Input. There are still a bunch of tests that I want to write to have a better coverage of things, but having something is way better than nothing. I'm hoping the smoke tests will reduce the amount of manual testing I'm doing, too.
Vagrant: I took some inspiration from Erik Rose and DXR and wrote a Vagrant provisioning shell script. This includes a docs overhaul as well. This work is almost done, but needs some more testing and will probably land in the next week or two. This will make peoples' lives easier.
Automated translation system (human and machine): I wrote an automated translation system. It's generalized so that it isn't model/field specific. It's also generalized so that we can add plugins for other translation systems. It's currently got plugins for Dennis, Gengo machine translation and Gengo human translation. I turned the automated human translation on yesterday and it seems to be working well. That was a HUGE project. I'm glad it's done.
One thing it includes is a lot of auditing and metrics gathering. This will make it possible for me to go back in time and look at how the translation system worked on various Input feedback responses and hone the system going forward to reduce the number of human translations we're doing and also reduce the number of problems we have doing them.
Better query syntax: We were upgraded to Elasticsearch 0.90.10. I switched the query syntax for the dashboard search field to use Elasticsearch simple_query_string. That allows users to express search queries they weren't previously able to express.
utm_source and utm_campaign handling: I finished the support for handling utm_source and utm_campaign querystring parameters. This allows us to differentiate between organic feedback and non-organic feedback.
More like this: I added a "more like this" section to the response view. This makes it possible for UA analyzers to look at a response and see other responses that are similar.
Dashboards for you, dashboards for everyone!
I'm putting this in its own section because it's intriguing. I'll write another blog post about it later in July as things gel.
On Thursday, a couple of days after d3 training that Matt organizied, I threw together a better GET API for Input feedback responses. It's not documented, it probably has some bugs and it's probably going to change a bit, but the gist of it is that it lets you more easily build a dashboard that meets your needs against live Input data.
Here's a proof-of-concept:
That's looking at live Input data using the new GET API. The code is in a GitHub gist. It auto-updates every 2 minutes.
The problem is that I've got a ton of Input work to do and I just can't write dashboard code on Input fast enough. Further, of the people I've talked to that use the front page dashboard, they all have really different questions they're asking of the data. I'm hoping this alleviates that bottleneck by letting you and everyone else write dashboards that meet your needs.
I encourage you to take my proof-of-concept, fork the gist, tweak it, use bl.ocks.org or something to "host" the gist. Build the dashboard that answers your questions. Share it with other people. Plus, let me know about it. If you have issues with the API, submit a bug and tell me.
If this scratches the itch I think needs scratching, it should result in a bunch of interesting dashboards. If that happens, I'll write some code in Input to create a curated list of them so people can find them more easily.
This was a really crazy quarter and parts of it really sucked, but we got a lot accomplished and we laid some groundwork for some really interesting things for 2014q3.
Mon, 23 Jun 2014
I publish a status report to the input-dev mailing list every couple of weeks or so covering what was accomplished and by whom and also what I'm focusing on over the next couple of week. I sometimes ruminate on some of my concerns. I think one time I told a joke.
Since the last report:
Landed and deployed:
Landed, but not deployed:
- c348989 Add bug triaging for new contributors section
- 5b7dc67 Add Gengo API tests and skip_if infrastructure
- 98d30fb [bug 1026131] Add Gengo human translations bookkeeping
- 38d8584 [bug 1026131] Rework translations system logging code
- 1d9e67a [bug 1027293] Add audit records to response view
Mostly I spent the last couple of weeks working on automated Gengo human translation support. This involved some infrastructure rewriting plus some additional infrastructure so that when we push all this out, we can see what's going on as it is happening.
Additionally, I went through and updated the mentor metadata for mentored bugs, added a bunch of new mentored bugs and worked with two potential contributors on them.
Over the next week (last week in 2014q2):
- finish up automated Gengo human translation work
First thing in 2014q3, I'll spend some time "opening up" the development side of the project. This will make it easier/possible to follow and participate in development. I'm still figuring out some of the details and it's likely I'll continue to change how things work over the course of the quarter, but plan to follow advice from the Community Building team and Erik Rose who seems to be doing really super with DXR.
In my previous post, I mentioned how Bugzilla grew a mentor field and had some code on how to query Bugzilla using the old and new APIs for the list of mentored bugs. This updates that information.
Gerv and Byron pointed out there's a isnotempty operator that's better to use than the way I cobbled together to query for bugs that have data in the bug_mentor field.
Thus, the code should look like this:
import requests # Using the old BzAPI: https://wiki.mozilla.org/Bugzilla:REST_API r = requests.get( 'https://api-dev.bugzilla.mozilla.org/latest' + '/bug?product=Input&f1=bug_mentor&o1=isnotempty' ) data = r.json() print len(data['bugs']) # Prints 9 # Using the new BMO API. https://wiki.mozilla.org/BMO/REST r = requests.get( 'https://bugzilla.mozilla.org/rest' + '/bug?product=Input&f1=bug_mentor&o1=isnotempty' ) data = r.json() print len(data['bugs']) # Prints 9
Thu, 19 Jun 2014
- Updated June 23rd, 2014:
- There's a better way to query the data. See the update blog post
Bugzilla grew a mentor field recently. This is really fantastic as it solves some interesting problems and makes it easier to track various aspects of mentoring which have been previously difficult to track. Yay to everyone involved in making that happen!
Migrating from the old way (sticking [mentor=xxx] in the whiteboard field) to the new way caused a problem that I spent a while working on today. I heard reports of other people having the same problem, hence this blog post.
There are a bunch of Bugzilla-symbiotic systems which would show a list of mentored bugs by checking to see if the string mentor= was in the whiteboard field. That no longer works. Instead we have to check to see if the bug_mentor field is empty. However, this is difficult to express with the old Bugzilla REST API (BzAPI).
The bug_mentor field is unique in that it holds email addresses which have the @ in them. So we can (ab)use this property by seeing if the bug_mentor field contains the @ character.
Here's some Python that shows this with the old BzAPI and the new BMO API which pulls mentored bugs for Input:
import requests # Using the old BzAPI: https://wiki.mozilla.org/Bugzilla:REST_API r = requests.get( 'https://api-dev.bugzilla.mozilla.org/latest' + '/bug?product=Input&bug_mentor_type=contains&bug_mentor=@' ) data = r.json() print len(data['bugs']) # Prints 9 # Using the new BMO API. https://wiki.mozilla.org/BMO/REST r = requests.get( 'https://bugzilla.mozilla.org/rest' + '/bug?product=Input&bug_mentor_type=substring&bug_mentor=@' ) data = r.json() print len(data['bugs']) # Prints 9
Tue, 13 May 2014
Better search syntax is here!
Yesterday I landed the changes for bug 986589 which affects all the search boxes and search feeds on Input. Now they use the Elasticsearch simple-query-search query instead of the hand-rolled query parser I wrote.
This was only made possible in the last month after we were updated from Elasticsearch 0.20.6 (or whatever it was) to 0.90.10.
Tell me more about this ... syntax.
I'm pretty psyched! It's pretty much the minimum required syntax for useful searching. It's kind of lame it took a year to get to this point, but so it goes.
To quote the Elasticsearch 0.90 documentation:
+ signifies AND operation | signifies OR operation - negates a single token " wraps a number of tokens to signify a phrase for searching * at the end of a term signifies a prefix query ( and ) signify precedence
Negation and prefix were the two operators my hand-rolled query parser didn't have.
What does this mean for you?
It means that you need to use the new syntax for searches on the dashboard and other parts of the site.
Further, this affects feeds, so if you're using the Atom feed, you'll probably need to update the search query there, too.
Also, we added a ? next to search boxes which links to a wiki page that documents the syntax with examples. It's a wiki page, so if the documentation is subpar or it's missing examples, feel free to let me know or fix it yourself.
Wed, 09 Apr 2014
I have this basic problem where I do a lot of web-site work and I need to show people what I've done so far so they can review it and help me make it better or make it suit their needs better. Screenshots aren't very helpful because the site is interactive. Further, the site needs to get tested on multiple devices/platforms/browsers. Also, I need to make sure that the site is only accessed via https.
What I've been doing up to now is failing miserably: I'd push work to our staging server for people to test out, but that sucks as an answer and affects my co-workers and makes a mess of our staging server. Plus iterating on things is difficult.
- endpoint must be https-only
- must be easy to set up and take down
- must be easy to access so people can easily test things on my local machine
I looked around and this would be pretty easy to do if I didn't have the https-only requirement. That makes things difficult without a lot of work.
Then I found pagekite. They make it really easy.
Here's how you set it up:
Download and install the pagekite software: http://pagekite.net/downloads/
Run your website. In my case, I'm working on Django sites, so I launch like this:
$ ./manage.py runserver
That runs the Django project I'm working on on localhost:8000.
$ pagekite.py 8000 YOUR_NAME.pagekite.me:443
That creates a tunnel from your machine to the pagekite.me server. When someone accesses https://YOUR_NAME.pagekite.me/, the request goes through the tunnel to your pagekite backend and that performs the request over http to your local webserver (in my case, the Django project) bound to localhost:8000.
Access is https-only. If anyone tries to access http://YOUR_NAME.pagekite.me/, then they get a connection error.
The https-only requirement is satisfied by restricting the kite to only listening to port 443--the https port. That's pretty key.
This lets me run my Django project locally on http without dealing with self-signed certificates, but still require https access so data isn't floating around in clear text.
The one problem with this is that my local server thinks it's running http and so redirects that include the protocol go to http rather than https.
If you don't already have an account, I'm pretty sure step 3 will walk you through setting one up. Free accounts are limited in what they can do.
Also, they hang out on #pagekite on Freenode. I had a problem, asked a question and got a super helpful reply. The code is Open Source, so it's possible to look through it and debug it.
I'll be using this going forward.
Why write this?
This is a common use case for web developers. I figured I'd write this up because the https-only part is pretty key and it was the part that I had to ask for help with.
Thu, 03 Apr 2014
What is it?
ElasticUtils is a Python library for building and executing Elasticsearch searches.
See the Quickstart for more details.
This is a big release, but there are some compromises in it that I'm not wildly excited about. Things like Elasticsearch 1.0 support didn't make the cut. I'm really sorry about that---we're working on it.
This release has a lot of changes in it. Roughly:
- dropped pyelasticsearch for elasticsearch-py (Thank you Honza!)
- fixed S.all() so it does what Django does which should let you use an S in the place of a QuerySet in some cases
- new FacetResult class (Thank you James!)
- S.facet() can take a size keyword
- cleaned up ESTestCase
- SearchResults now has facet data in the facets property
For the complete list of what's new, What's new in Version 0.9.
Many thanks to everyone who helped out: Alexey Kotlyarov, David Lundgren, Honza Král, James Reynolds, Jannis Leidel, Juan Ignacio Catalano, Kevin Stone, Mathieu Pillard, Mihnea Dobrescu-Balaur, nearlyfreeapps, Ricky Cook, Rob Hudson, William Tisäter and Will Kahn-Greene.
We're going to be sprinting on ElasticUtils 0.10 at PyCon US in Montreal mid April. If you're interested, come find me!
If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.
Fri, 20 Dec 2013
It was a big year for SUMO. In 2012, we got a lot accomplished: new search, new information architecture,
One thing I didn't do was make my year-end script product better output.
Anyhow---on with stats!
Twas the year: 2013 =================== Bugzilla ======== Bugs created: 889 a.topal : 156 rrosario : 100 willkg : 87 scoobidiver : 85 ibai : 58 mverdi : 54 mcooper : 42 feer56 : 35 krystaiceman : 26 rdalal : 19 david.weir : 15 shuhao : 13 swarnavasengupta : 9 andrei.hutusoru : 8 me+bugzilla : 7 tobbi.bugs : 6 mluna : 6 joshua-smith : 6 tdowner : 6 leszekz : 6 yoshi.yokotani : 5 stephen.donner : 5 slurp : 5 mana : 5 madperson : 4 kbrosnan : 4 tonnes.mb : 4 rardila : 4 pmcclard : 4 dbialer : 4 michaljev : 4 abc : 4 l10n : 4 pcvrcek : 3 rdaub : 3 fabricio : 2 rmcguigan : 2 sudheesh1995 : 2 alex_mayorga : 2 simone.lando : 2 nishant_cs : 2 bram : 2 smolejv : 2 bob.silverberg : 2 rtanglao : 2 kdurant35rules : 2 amit103065 : 2 subedimahadev : 2 lhenry : 2 thomas.lendo : 2 shawnsumo : 2 mhammond : 1 kdurant35rules : 1 djst : 1 curtisk : 1 chiorean.ioana : 1 bermea : 1 friedel : 1 bputstudentweb : 1 margaret.leibovic : 1 rbillings : 1 nikitan.dolmart : 1 georgevidalakis : 1 nsm.nikhil : 1 satishb3 : 1 bwbrowning : 1 bugzilla : 1 coce : 1 EddyCarr : 1 gryllida : 1 mohammed.samad : 1 6a68 : 1 krupa.mozbugs : 1 John99-bugs : 1 wjohnston : 1 barderne : 1 jan0286 : 1 fwenzel : 1 rnewman : 1 this4midhun : 1 bjohnson : 1 bugzilla-fromthedeep : 1 iamjithin : 1 bmo2010 : 1 chrismore.bugzilla : 1 evold : 1 jbertsch : 1 yousef : 1 pmjcreations : 1 rhelmer : 1 danishka : 1 mail : 1 gphemsley : 1 Rebeccah : 1 ckreinbring : 1 stephen : 1 berker.peksag : 1 jezdez : 1 nchen : 1 iamjayakumars : 1 netfuzzerr : 1 benjamin : 1 Bugs resolved: 1116 rrosario : 386 resolved, 273 fixed rdalal : 152 resolved, 150 fixed a.topal : 121 resolved, 41 fixed mcooper : 118 resolved, 105 fixed willkg : 72 resolved, 61 fixed scoobidiver : 31 resolved, 0 fixed michaljev : 20 resolved, 17 fixed swarnavasengupta : 18 resolved, 0 fixed shuhao : 16 resolved, 13 fixed me+bugzilla : 12 resolved, 5 fixed berker.peksag : 12 resolved, 12 fixed mverdi : 11 resolved, 5 fixed erik : 9 resolved, 9 fixed lhenry : 8 resolved, 1 fixed krystaiceman : 7 resolved, 1 fixed tobbi.bugs : 7 resolved, 6 fixed joshua-smith : 6 resolved, 4 fixed jfong : 5 resolved, 5 fixed tdowner : 5 resolved, 2 fixed feer56 : 5 resolved, 0 fixed david.weir : 5 resolved, 4 fixed bwbrowning : 5 resolved, 5 fixed bram : 5 resolved, 4 fixed ibai : 4 resolved, 1 fixed alastra.mariagrazia : 4 resolved, 2 fixed laura : 4 resolved, 4 fixed williamr : 3 resolved, 1 fixed buchanae+bugs : 3 resolved, 3 fixed bharath_ves : 3 resolved, 3 fixed paul+moz : 3 resolved, 3 fixed chris.lonnen : 2 resolved, 2 fixed smolejv : 2 resolved, 1 fixed zcampbell : 2 resolved, 0 fixed iamjayakumars : 2 resolved, 0 fixed curtisk : 2 resolved, 0 fixed leszekz : 2 resolved, 0 fixed abc : 2 resolved, 1 fixed pcvrcek : 2 resolved, 0 fixed taygunagiali : 2 resolved, 2 fixed mail : 1 resolved, 0 fixed bmo2010 : 1 resolved, 1 fixed wymette : 1 resolved, 0 fixed beaotx : 1 resolved, 1 fixed nelson : 1 resolved, 1 fixed madperson : 1 resolved, 1 fixed coce : 1 resolved, 0 fixed pmcclard : 1 resolved, 1 fixed tgavankar : 1 resolved, 1 fixed bugzilla-fromthedeep : 1 resolved, 0 fixed rtanglao : 1 resolved, 0 fixed stephen.donner : 1 resolved, 1 fixed guillermo.movia : 1 resolved, 1 fixed lorchard : 1 resolved, 1 fixed nukeador : 1 resolved, 0 fixed rtucker11 : 1 resolved, 0 fixed nishant_cs : 1 resolved, 0 fixed stas : 1 resolved, 0 fixed mattbasta : 1 resolved, 1 fixed satishb3 : 1 resolved, 0 fixed ragsagar : 1 resolved, 1 fixed rmcguigan : 1 resolved, 1 fixed nchen : 1 resolved, 0 fixed kudrom : 1 resolved, 1 fixed andrei.hutusoru : 1 resolved, 0 fixed reed : 1 resolved, 0 fixed tiziana.sel : 1 resolved, 0 fixed chance.zibolski : 1 resolved, 1 fixed alice0775 : 1 resolved, 0 fixed ravi : 1 resolved, 0 fixed nsm.nikhil : 1 resolved, 0 fixed gryllida : 1 resolved, 1 fixed deletesoftware+moz : 1 resolved, 0 fixed pmjcreations : 1 resolved, 0 fixed boerni : 1 resolved, 1 fixed rforbes : 1 resolved, 0 fixed dbialer : 1 resolved, 1 fixed jgross : 1 resolved, 1 fixed INCOMPLETE : 43 WONTFIX : 51 DUPLICATE : 64 INVALID : 73 WORKSFORME : 121 FIXED : 764 Research bugs: 15 761582: [research] Add feature: Articles that link to this article 788104: [research] [ux] Support multiple products in the support forum 815089: [research] Investigate telling apart Firefox for Desktop and Firefox for Mobile tweets 816970: [research] SurveyGizmo API to be used in automated exit survey 823060: [research] Use datetime instead of ints in ES mappings 823891: [research] Adding KB revisions feature 825621: [research] Store the templates, article links and images in each article 825624: [research] Investigate how to update to Twitter API v1.1 841412: [research] Bad localization strings shouldn't break the site. 845290: [research] URL bar should fade away on SUMO 854554: [research] Youtube embeds don't work with templates 889884: [research] Open Badges! 889890: [research][discuss] figure out how to improve our l10n situation with search 906992: [research] Add support for multiple ES indexes by doc type 937889: [research] Login users via a URL in email Tracker bugs: 20 433161: [Tracker] Support for forums in other languages 625891: [tracker] HTML email 721462: [tracker] Taxonomy IA improvements 758598: [Tracker] Search UX suggestions 783262: [tracker] Add rate limiting to protect us from spammers 790785: [Tracker] L10n tools editing part 790786: [Tracker] L10n tools organization part 800962: [Tracker] Add activity history page for KB 815625: [Tracker] Segment dashboards and other contributor pages by product 817540: [tracker] AJAXify the refine+focus panel 825606: [tracker] Switch everything from Webtrends to Google Analytics 827640: [tracker] Localize Questions 838584: [Tracker] Getting ready for Firefox OS launch 845286: [tracker] Use as little bandwidth as possible on mobile version of SUMO 845773: [Tracker] move to an OS charting solution 848520: [tracker] Make all traffic HTTPS 851730: [tracker] Close threads pro-actively 871559: [tracker] update codebase to django 1.4 layout 897057: [tracker] Open Badges -- stage 1 920530: [tracker] support Webmaker on SUMO git === Total commits: 1138 Ricky Rosario : 492 (+16258, -16435, files 2972) Will Kahn-Greene : 178 (+8311, -3748, files 438) Rehan Dalal : 168 (+13016, -5554, files 680) Mike Cooper : 145 (+46955, -22136, files 582) Kadir Topal : 39 (+352, -110, files 61) Michał Frontczak : 19 (+229, -182, files 78) Berker Peksag : 15 (+570, -717, files 73) Shuhao Wu : 15 (+1523, -127, files 51) Jen Fong-Adwent : 9 (+138, -18, files 17) Tobbi : 8 (+338, -204, files 13) browning : 5 (+140, -16, files 12) davd Weir : 4 (+15, -1, files 4) Joshua Smith : 4 (+94, -87, files 13) Tobias Markus : 3 (+8, -8, files 4) Anush : 3 (+4, -1, files 3) Gaurav Dadhania : 3 (+3, -3, files 3) Bharath Thiruveedula : 3 (+15, -14, files 3) ibai : 3 (+30, -30, files 4) kudrom : 2 (+9, -9, files 5) Nghi Tran : 2 (+2, -1, files 2) Tanner Filip : 2 (+4, -4, files 2) Börni : 2 (+30, -15, files 4) madperson : 2 (+5, -4, files 2) Taygun AGIALI : 2 (+7, -6, files 3) TylerDowner : 2 (+3, -3, files 2) James Socol : 2 (+37, -27, files 3) david-w : 1 (+1, -1, files 1) ragsagar : 1 (+16, -1, files 2) Guillermo Movia : 1 (+1, -0, files 1) rosanaar : 1 (+9, -0, files 1) Gryllida : 1 (+26, -6, files 3) Beatriz Nombela : 1 (+9, -9, files 6) Total lines added: 88158 Total lines deleted: 49477 Total files changed: 5048
Ricky does a lot of work! Holy cow!
In 2011, we had 19 people who contributed code changes.
In 2012, we had 23 people.
In 2013, we had 32 people.
Like 2011 and 2012, we resolved more bugs than we created in 2013. That's three years in a row! I've never seen that happen on a project I work on.
There are a lot of people braving Bugzilla to write up bugs. Skimming the list, I see developers, non-developers, Support contributors, localizers, support team and a lot of people I don't recognize.
Here's some number comparisons:
I spent a good chunk of 2013 working on Input, but here's what I remember from SUMO development in 2013:
- We rearranged the codebase for better Django 1.4 layout. That was a project. Oy.
- We added support for non-English languages to the support forums!
- We switched email to be HTML formatted. We also reworked email to be localized.
- We switched to Google Analytics.
- We implemented Open Badges---though there's still a few important pieces to finish there.
- We switched to YouTube for videos.
- We added support for Webmaker and Firefox OS. Thunderbird support will be added to SUMO in 2014.
- Mike took a lantern, a crust of bread and a big sword and spelunked into the darkest dungeons filled with stinky, squelchy muck and rewrote the showfor code.
- We reworked our search code to handle multiple indexes, though we haven't taken advantage of that, yet.
- We switched deployment to use Dennis to lint all translated strings before pushing them to production. This has almost assuredly saved us from production fires. I hated those kinds of fires. Hooray for Dennis!
- We wrote and switched to Ernest for sprint planning and coordination.
- We overhauled everything to add support for Persona authentication, but had to push off deployment indefinitely because of problems with Persona which are being ironed out by the Persona team.
- We added an escalation system for questions that haven't received a response in x hours for some positive value of x that is still in flux.
- We ditched Highcharts.
- We wrote a command-line deployer which tells us exactly what's going out and tells New Relic, too. This gives us a much better idea of what we're deploying and how it affected the site afterwards. This command-line deployer is named chief-james in honor of James who has moved on to greener and well measured pastures.
- We added a bunch of new metrics, dashboards, history pages, activity pages, icons, bicons, landing pages, take-off pages, topics, subtopics, toe picks and all kinds of stuff.
That's the gist of the year: it was a lot of work, but we accomplished a ton.
w00t for 2013!
Thu, 19 Dec 2013
It was a big year for Input. In 2012, we spent the last half rewriting Input. In 2013, it went through secreview, had a bunch of things fixed and then we migrated to the new system.
Since then, we've been fixing bugs, reimplementing features that were lost and writing the scaffolding for the new set of User Advocacy dashboards and tools.
Let's look at some Bugzilla and git stats for the year:
Twas the year: 2013 =================== Bugzilla ======== Bugs created: 150 willkg : 100 cwwmozilla : 5 fbraun : 4 mgrimes : 4 tdowner : 3 stephen.donner : 3 me+bugzilla : 2 gasell+mozilla : 2 mcooper : 2 glind : 2 mozaakash : 1 kdurant35rules : 1 hitmanarky : 1 kbrosnan : 1 bob.silverberg : 1 splewako : 1 rrosario : 1 mattbasta : 1 educmale : 1 feer56 : 1 326374 : 1 anthony : 1 shopov.bogomil : 1 peterbe : 1 l10n : 1 chrismore.bugzilla : 1 landis : 1 dron.rathore : 1 rq : 1 MattN+bmo : 1 joshua-smith : 1 cturra : 1 swagat.kanungo : 1 Bugs resolved: 268 willkg : 157 : WONTFIX 50 : FIXED 89 : WORKSFORME 8 : DUPLICATE 9 : INVALID 1 cwwmozilla : 57 : FIXED 1 : WONTFIX 7 : WORKSFORME 29 : DUPLICATE 1 : INVALID 19 mgrimes : 10 : FIXED 1 : DUPLICATE 1 : WORKSFORME 5 : INVALID 3 shopov.bogomil : 7 : WONTFIX 1 : WORKSFORME 2 : INVALID 1 : FIXED 2 : DUPLICATE 1 mcooper : 6 : DUPLICATE 1 : FIXED 5 mozilla : 5 : FIXED 5 me+bugzilla : 4 : WONTFIX 1 : FIXED 1 : DUPLICATE 1 : INVALID 1 mozaakash : 2 : WORKSFORME 1 : INVALID 1 trifandreialin : 2 : WORKSFORME 2 rrosario : 2 : FIXED 2 joshua-smith : 2 : FIXED 1 : INVALID 1 aaron.train : 2 : WONTFIX 1 : DUPLICATE 1 stephen.donner : 1 : INCOMPLETE 1 emorley : 1 : FIXED 1 curtisk : 1 : INVALID 1 unghost : 1 : WORKSFORME 1 rajul.iitkgp : 1 : FIXED 1 jruderman : 1 : INCOMPLETE 1 chris.lonnen : 1 : FIXED 1 nigelbabu : 1 : FIXED 1 tofumatt : 1 : FIXED 1 cturra : 1 : FIXED 1 fwenzel : 1 : FIXED 1 mbrandt : 1 : FIXED 1 INCOMPLETE : 2 DUPLICATE : 15 INVALID : 28 WORKSFORME : 48 WONTFIX : 60 FIXED : 115 git === Total commits: 277 Will Kahn-Greene : 249 (+51602, -16851, files 1130) Mike Cooper : 11 (+38528, -236, files 217) Brandon Burton : 7 (+42, -215, files 9) Ricky Rosario : 4 (+36, -19, files 6) Bob Silverberg : 3 (+19, -9, files 3) Rajul : 1 (+3, -0, files 1) Joshua Smith : 1 (+10, -5, files 1) bogomil : 1 (+1, -1, files 1) Total lines added: 90241 Total lines deleted: 17336 Total files changed: 1368
I want to highlight some interesting bits:
We resolved more bugs than we created. That's partially due to us going through and closing out old bugs for the old Input that aren't relevant anymore.
According to the Bugzilla and git data, there were 47 contributors to Input this year: 326374, Bob Silverberg, Brandon Burton, Joshua Smith, MattN+bmo, Mike Cooper, Rajul, Ricky Rosario, Will Kahn-Greene, aaron.train, anthony, bogomil, chris.lonnen, chrismore.bugzilla, cturra, curtisk, cwwmozilla, dron.rathore, educmale, emorley, fbraun, feer56, fwenzel, gasell+mozilla, glind, hitmanarky, jruderman, kbrosnan, kdurant35rules, l10n, landis, mattbasta, mbrandt, me+bugzilla, mgrimes, mozaakash, nigelbabu, peterbe, rajul.iitkgp, rq, splewako, stephen.donner, swagat.kanungo, tdowner, tofumatt, trifandreialin, and unghost.
That doesn't include localizers who do a ton of work translating the strings in the Input ui.
That includes some of the folks who work on the input-tests repository, but possibly misses some.
Most of the 47 contributors are not "core developers". That's cool, but I could be doing a better job here making it easier for non-core developers.
Those are the stats.
At a high-level, we accomplished the following:
- stood up a new Input code base
- the beginnings of spam identification and removal
- Input API for feedback submission
- Firefox OS feedback form
- infrastructure for an Analysts group with special privileges
- the beginnings of an Occurrence Comparison report dashboard
One thing I discovered in 2013q4 was that it's really hard to be the mostly-solo dev on a project like this. I'm lucky that I'm part of a larger team, so peer reviews for work I've done is possible and timely. However, I find I'm switching contexts between the technical details of what I'm working on now and the high-level details of a bunch of possible future tasks/projects. That's really hard to do day-to-day and still maintain development momentum. I have some thoughts on how to serialize my work so that I'm doing less context switching and I can focus on individual things more deeply which should produce better outcomes.
My goals for Input for 2014 are these:
- clean up the code base: there's still a bunch of weird stuff in there from the rapid development work we did in 2012
- reduce barriers to entry for new contributors: better documentation, fewer steps to get up and running, more bugs marked for mentoring, more outreach, ...
- build infrastructure that we can use for better User Advocacy tools: watched alerts, email notifications, dashboards, ...
- flesh out tests: we're really light on smoketests and regression-catching tests
- work with Matt and Cheng to figure out where Input fits into the grand scheme of things; how can we make it a general-purpose feedback system? how can we handle non Firefox products and initiatives?
Yay for 2013!
My script only showed top tens which misses tons of people who did work. I redid the data and that increases the number of contributors from 16 to 47. Oops!