Friday, 5 August 2016

A Docker Experiment

Containers are a topic that have been rising in occurrence for me, for the last year or so. From using them as part of the architecture at work or for various pet projects friends have been working on. I figured it was time to experiment myself and get to grips with what people have been talking about.
It seemed like a good idea to find some introductory material that would give me an overview of its uses without delving too far into the details so I could see what it actually was, so I found a PluralSight course about Docker, specifically “Docker and Containers:The Big Picture”. This gave a nice overview of what Docker is, and more importantly what it was trying to achieve and how to use it.
With a bit more of an understanding, I wanted to use it for something, preferably something familiar. I decided to try setting up ElasticSearch and Kibana containers, where Kibana would visualize the ElasticSearch data.
I used bits of this article along the way as a guide, if you'd prefer a more detailed reference:
If you're on Windows 10 you might have a slightly different experience as you're able to use Docker for Windows. The machine I had available is on Windows 8 so I used the Docker ToolBox, so this post will assume you're doing similar.
At installation, if you don't have Hyper-V you'll want to tick the box for installing VirtualBox as you'll need one or the other, you can also useVMWare, etc. but I'm using Hyper-V. If you've not used virtualisation on your system before you may need to turn on something like VT-x to continue. You will also require a virtual switch. If you're a Hyper-V user on a laptop you might want to read on a bit first, as I encountered a problem here.
With that installed I opted to use PowerShell to continue. Mostly because it looks better in screenshots. It needs to be elevated to Administrator to use Hyper-V.
With setup out of the way you want to choose a name for your docker machine and run something like:
    docker-machine create --driver hyperv elkmachine

If you're not using Hyper-V you'll need to change the driver to whichever you've decided to use, and replace elkmachine with your chosen machine name.
This will create a docker virtual machine that you can see in the Hyper-V Manager.
You may run into a problem where it seems to hang at 'Waiting for host to start'. I found this when I attempted it on my laptop initially, and came across this post:
Simply put, a standard virtual switch won't work, and you need to create an internal virtual switch and in the properties of your network connection, share the connection with the WiFi virtual switch. In addition to the article, if you have a regular virtual switch and a WiFi virtual switch, docker create might default to the first one. This could be the order that they're created in but I didn't test it, and just removed the external virtual switch as it probably wouldn't work anyway.
With that sorted, you should see something like this:

If you look in Hyper-V now, you'll see a virtual machine with the docker machine name with some boot2docker text in the window.

The docker machine's now been created, we need to use set env to create containers on it.

As the terminal says we need to run:
    "C:\Program Files\Docker Toolbox\docker-machine.exe" env elkbox | Invoke-Expression
Somewhere around or before this point you'd want to decide what container you want to run, there's an excellent resource of existing containers for most mainstream software at There are other alternative sources, but lucky for us, both ElasticSearch and Kibana have official containers here. Before we can use the containers we need to 'pull' them. This is on a per-machine basis, so each of your docker machines will need to pull them separately, but within a machine once you've pulled a container it's usable.
First we want to pull ElasticSearch:
    docker pull elasticsearch
Following that we want to run the container, with some flags specified, the command we will use is:
    docker run -d -p 9200:9200 -p 9300:9300 elasticsearch
The -d will run the container in detached mode, particularly useful with ElasticSearch which will otherwise block the terminal.
-p maps a container port to a host port. In these cases we want the mapping to be the same but this is useful for duplicate containers.
Once this is done we can check its status with "docker ps", you can see the result of the previous commands below.

We now have an ElasticSearch container running, but we should check on it further. To hit it we need to know the IP address of the docker machine. We can do this by using:

    docker-machine ip elkbox

When we ran the container we set ports 9200 and 9300, so with the IP address and the port, we can hit ElasticSearch from the browser and should get some JSON back to let us know it's there, like so:

So, we know ElasticSearch is ready to go, now we want to set up a Kibana container and point it at ElasticSearch. Like ElasticSearch, we need to pull the container, which is simply:

    docker pull kibana

After some fiddling I found the best way to run the Kibana container was with this command:

    docker run –name kibana -e ELASTICSEARCH_URL= -p 5601:5601 -d kibana

You can see once again we're mapping some ports, and we're using one of the containers parameters to set the URL of ElasticSearch for it to use. Once again, we can test it easily as we know it's at the same IP but on port 5601 as we specified in the run command. And in the browser you should see something like:

Now it's just a case of pushing your data to ElasticSearch. I'm finding this approach preferable to my previous way of running these 2 services in command prompts, since I don't have to have a bunch of windows open and I can stop the docker machine and not have to worry about remembering specifically which services I've started as they all belong to the same docker machine. 
It was also quicker to get going immediately even with the docker installation when considering installing Java and potentially other prerequisites. I'll be doing some more experimentation around this soon.

Monday, 13 June 2016

Self-Producing an EP Pt.6: Mastering

While everything was being mixed into the same chain of plug-ins individually, we did at least know that mastering should be a clearly defined separate step. Again, we considered getting it done professionally, but sort of figured if we've made a mess we may as well roll with it, and if we haven't then maybe this won't go so horribly either.
It didn't take long to get this part done comparatively, maybe a few evenings. Having tried out a large amount of presets from various plugins I had a good idea how I wanted it to sound. All the mixed tracks were rendered, aiming for their peak to be around -6db, and dropped into one project. The preset from the tracks was applied, and then a run-through to listen for consistency in volume, sound, etc.
Oh dear. Two of the tracks appeared to have almost no bass in them, while the other three were quite consistent with each other. That had to be fixed and took an evening to identify and correct some high-pass filtering that was too aggressive on the guitars, and low bass guitar volumes. They'd sounded fine on their own, but stood up next to the others were clearly lacking. There was also some fizzing making one of the tracks sound quite bad too, which also got cleared up. Then it was a case of getting the mastering plugins right to allow for the right volume when compared to a reference track, and then varying each individual songs output volume to match the perceived volume.
I'd like to say that was it, but after we had a few reviews of all the tracks we had a handful of issues that were going to need adjusting. Chorus vocals slightly too loud in one place, bass guitar too quiet in another, and a couple of others that required some mix changes, and then tweaking the mastering again to get the consistency back.

Things that were important:

  • It's good that there was a (after that initial bit of repair) consistency across all the tracks, I think the effects chains I made mixing the first song helped a lot here, and having a mastering pre-set applied.
  • Doing them all in one project, as you'd expect, helped put them all together.
  • Using a reference track to check where the perceived volume was good.
  • We didn't destroy all of the dynamics, which we said we wanted to be the case from the start.
  • Maybe get someone else to do it next time, but we'll see.
  • Try and identify the handful of mixing problems before this point so there's no painful back and forth of rendering new mixes.
  • It doesn't sound amazing in mono. It was referenced in mono a fair bit but it's heavily reliant on stereo for space. I'm not sure this is much of a problem but worth considering.

At this point the audio part is mostly complete, but with the intention of making a 'radio edit' as it were of 1 or 2 of the tracks (trim some extended sections and make sure it works in mono, probably) to submit to various places there's still a bit to be done. There is of course now all the other stuff that you need to do. Photos, a logo, all the social media stuff, web-sites, uploading for streaming, and attempting to book gigs. I might post about those when I've actually done them.

Monday, 6 June 2016

Self-Producing an EP Pt.5: Mixing

After some discussion we opted to mix it ourselves for a few reasons.

  1. It's our first EP, it's not going to be heard by a huge amount of people and while it's important that it sounds good, it's expensive to get done properly and we might be able to manage 'good enough' on our own with a lot of work.
  2. Mixing as an online service seems to be the main way to go for small/starting out bands and that seems too creatively detached. There are many stories floating around of people who've sent things off for mixing and the mixer has a much different idea of how it sounds.

Maybe it's not the best choice, but it seemed an acceptable risk that we would at least attempt to have it 'look' how we want and it be a bit wonky than potentially end up with something we're not happy with in a different way. The benefit would have been being able to just ship the stems off and make it someone else's problem instead of weeks of being unsure while swimming in these unfamiliar waters. Someone with professional level skills. Maybe next time.

The most blindingly obvious thing here is that the recording stage should have finished before this stage begun. It restarted a few times due to what was ultimately just me being unhappy with the vocals. What eventually happened after a few false starts was the bit that should have been done before anyway. Editing. Some time was taken to look at the detail of the tracks and fixing some timing issues (but not all of them, just the important ones), removing in-breaths from the vocal tracks, making the volume consistent and some tuning. The first pass of tuning was way too heavy handed. I suspect this was as a result of me editing my own vocals. At the end of a pass of tuning they'd just sound robotic, and I'd end up undoing it all and only tweaking where necessary, and even then 'un-fixing' them later to stop the tweaks from making them perfect.

After the editing had been done the mixing started properly. Through the process of demo'ing and refining, the first track had got to a point where the mixing was mostly done and it was sounding pretty good. Some of this was from testing out general wisdom I'd found on the internet, such as removing bass from guitars/vocals to let it through on the instruments that needed it. I used this to make some track-based preset chains to apply to the tracks that weren't finished and use them as a starting point later and made a long list of things to do to prepare them all for the more detailed and contextual changes. The presets were helpful but needed changing more than I thought. New iterations would go into the bands Telegram channel for critiquing, and listening to on some different devices (good headphones, bad headphones, monitors, an iPod dock) to see where the problems were.

The EZMix mastering preset was removed and replaced with a chain of things I'd put together from Waves Gold (Black Friday Sale win) and some free plugins from the rest of the internet (FerricTDS and Molot, for starters) so everything was still being mixed into something consistent.
There were then a few weeks of successively spotting and hammering all the sticking out nails in. When you fixed the biggest, the next one appears. Eventually we ran out of nails (that we could percieve anyway) that seemed worth fixing. There might be some element of fatigue here having taken a few weeks or more at this point and I sent it to a handful of friends for critiquing further before we locked it down.

Things that were important:

  • The internet is a great resource for learning and I picked up a lot quite quickly.
  • Starting with presets that came with Reaper or Waves plug-ins and then tweaking/adjusting/playing to find what I wanted was quite educational.
  • There's a point where you have to leave it alone and the thing that seems like a blatant error to you after 100 hours of listening is completely inaudible to anyone else.
  • Recognising that even across 3 people there's variations in what they think it should sound like and ultimately 1 person should be driving.
  • Getting external feedback at the point where everyone is too fatigued to hear anything anymore.
  • Having task list and a set time frame (it slipped but it did at least force some focus) to complete by.
  • Given the time and the resources, we'd have someone else mix it, but incredibly selectively.
  • Make sure editing is completed and everything sounds good before getting to this point.
  • Do vocal tuning with context, what sounds bad with the track solo'd might be completely unnoticeable with everything else going on, and tuning it might even make it worse.
  • Avoid tracks that are completely out of genre while doing this, instead of hearing something by another band doing something entirely different and comparing them.

Monday, 30 May 2016

Self-Producing an EP Pt.4: Recording

We got to this point and as we'd done things the way we have, it wasn't really as cut-and-dry anymore where the demos ended and new recording sessions would begin. In the process of refining the demos, we'd recorded new tighter guitar and bass parts, and as the drums are programmed we just modified the midi to improve the demos. We considered re-recording the guitars but that seemed like a lot of work for no real gain. Some of the bass got recorded again but that was voluntary. Making the drums sound good was a bit different, that required moving to a different set of patches. EZDrummer is great and definitely serves it's purpose but sounded way too polished to be dropped in here. The input all stayed the same but we changed the patches around to find the sound we were looking for across all the instruments, and there was nothing more exciting to it than that.

Recording vocals however was going to require a bit more effort. The demo vocals weren't going to do, and they'd all need replacing. After a few experiments I opted for setting up my large condenser mic in a wardrobe that had been stuffed with pillows as insulation. The sound was clean enough and had the added benefit that I couldn't get too close to the microphone by mistake. After a few evenings of attempts I wasn't happy with and trying to do them piece by piece. I took a weekend and spent a couple of hours per track getting down 4-5 takes and comp'ing together the best take I could from each but not deleting any of the other takes on the off chance they were needed later. Around this point I experimented with triple tracking the parts in places where it was particularly heavy, to give it more depth. I couldn't quite make them sound right and I expect that's a combination of the mic placement and not being skilled enough to EQ out what I suspect was a lot of unnecessary frequencies building up to make it sound strange, so this idea got scrapped.

Things that were important:
  • Getting everything down in 2 sessions over 1 weekend helped with consistency.
  • Focussing on getting a decent comp and moving on stopped the on-going march of trying to improve them continuously.
  • It seemed like I'd had enough practise time to get it right, but I'd argue I definitely hadn't.
  • It might be a better idea to do this in a rehearsal or recording studio next time (ideally an actual studio, obviously) so things like volume levels aren't a concern when the neighbours are in. A proper studio has better suited mics, rooms and equipment and no noise concerns to worry about.

Monday, 23 May 2016

Self-Producing an EP Pt.3: Adding Vocals to Demos

After putting this off for a while, I finally got around to writing the vocal parts for the currently instrumental tracks. The delay was mostly around trying to work out how they were going to sound, stylistically. We knew they wouldn't be cookie monster vocals, but that was about it. It was just a case of working out what the melody was going to be for each section, putting some place holder words in that fit the rhythm and re-drafting them with some sort of theme in mind so they made at least some vague sense. Then they needed to go on the demos. An attempt was made to try and write a concept to have a theme, but as I re-drafted the lyrics, I was having to 'force' the theme into it, at which point I discarded it. The general idea remains in some of them but the detail has gone.

This bit was a pain. All the other instruments are either programmed or DI, so it's easy to make them sound good because we aren't mic'ing up amps or drums or anything like that. Now we can't do that, and it's hard to make it sounds like it belongs with the fairly tight demos when they're rough and badly mic'd. They didn't sound great, and at this point I'd never tried to fit vocals into a wall of noise like this before, just acoustic guitars. As they just needed to be indicative of what they might ultimately sound like I opted for over-processing them and rolling them in glitter. I tidied up the timing and the tuning and then drowned them in delay and reverb. Still not great, but they'd do.

Things that were important:

  • Getting something down at all so we can start critiquing and re-drafting and changing and cutting.
  • Drowning the demo vocals in effects and tuning to try and make them bearable.
  • Definitely start this earlier. It was blocking progress.
  • Aim for some consistency in recording them. Using 2 different mics in a handful of different locations on the same track made it distracting to listen to, and drew away from the goal of the actual music.
  • Accept that they're probably going to sound bad and get on with it.

Monday, 16 May 2016

Self-Producing an EP Pt.2: Rehearsals and Refining

Getting material to the point where it can be played and has some structure was followed by the obvious next step. Playing it. So we took 1 or 2 of the songs to a rehearsal room and started running through them to see if they worked or what they would sound like. We'd find out of the the demo track had too much of one particular instrument, or if it worked at all in the real world. The goal was to have something to perform live, which made this part important. We would come across issues that certainly needed fixing.
As there was no notation anywhere I would render mixes for each person that was particularly heavy on their instrument, and one with it completely missing, so that it was easier to hear the instrument as the focal point, and in context, and then have one to practise along to. This was also the point where other band members would take away my demo'd bass/drum parts and replace them with their own better ones. This could get messy quickly, so I kept mine as the master and requested stem/midi files back that I could drop into the master. In a previous iteration of the band we tried writing everything into Guitar Pro, but encountered the same problem with duplicate projects, and also nobody particularly wants to sit and write notation in midi for hours when they're trying to be creative.

Getting changes flying back and forth all the time was tricky to keep track of but kept momentum going. Progress is obvious when every couple of days there's a new version that has a new and improved bass part, or a better chorus, or a section removed entirely. Sometimes (rarely) the changes don't fit and you discuss it and it goes back to how it was or a new one comes in.

At this point I'll digress and state that this was is our first set of material, the egg in the chicken and egg situation of our goal of playing live. We're not a cover band, and to get gigs we need music, so we need to write and record some music to start playing. So it's not a band that's got material it can always reach for and can spend time meticulously crafting an album. The material should be good, but if we don't have a starting point, 2 years spent writing is 2 years wasted not playing. As result there was some blending between the recording and refining process.
In an ideal situation, in retrospect, we'd write the material and play it until we had no more changes, and then record it. But we opted for reality. In future, I suspect more time will be spent getting things right at this point rather than later.

Things that seemed important:

  • Getting into a regular practise schedule, especially when people have other commitments. If it becomes one of those commitments, then it's a lot easier to make progress.
  • Making it as easy as possible for everyone to know what they're supposed to be playing, and having the creative equality to make it their own.
  • Setting specific goals before heading into a practise session. eg. Make it all the way through 1 new demo.
  • Getting through the song in some form is better than not at all, we had no lyrics or vocals at this point and while it made it harder to keep track of the song progress in repetitive sections, having one less thing to worry about while playing them as instrumentals meant we got through them quicker.
  • Nobody remembers everything immediately, section charts with tempos and time signatures help keep everyone on the same page and help keep it together when it's still pretty rough.
  • Recording rehearsal session audio is a good idea for reflection and improvement but challenging in practise. Drums are loud, guitars that can be heard alongside drums are loud too. Everything is pointed in different directions, and finding the place to put a single microphone will take time. The other option is to go through the board and do something there. It's possible but will need some investigation.

Monday, 9 May 2016

Self-Producing an EP Pt.1: Demos

To jump right in, I started working with a few friends on an EP for our band. Now that it's coming to a close, I thought it might be a useful for me to document the process involved from the beginning, and useful information I've gathered along the way to help me get through things quicker and better on my next attempt at something like this.

At the point where I started piecing a few demo tracks together, I'd been using Reaper for a while and I was using EZMix and EZDrummer as my primary plugin set while making a few acoustic tracks. I subscribed to and and for a few minutes a day would have a dig through for whatever information was new.

This was pretty straight forward, I sat and tried out a few riff ideas (being primarily a guitarist), when something sounded good I'd record it into Reaper and apply a guitar preset to it, of which there are some great ones that just sound good straight away. I'd then track it again and pan hard (100% left for one and 100% right for the other) and then do some fixing on timing issues quickly.
Looping the riff, EZDrummer would be the next stop to find and tweak a drum loop that fit and drop it into the track directly. I preferred this to using the timeline in EZDrummer so I could just grab entire sections and copy/paste/move/delete easily without having to open it up and fiddle.
Somewhere in the middle of the guitar and drums getting put down I'd add one of the EZMix mastering presets to the master FX in Reaper. Something stylistically approximate. Then it's a case of grabbing the bass guitar and putting some bass down too.

To start putting a song together, I'd use the project as a bit of a sketchbook, and spend a few hours listening to what's there and just move along the timeline a bit and put some more ideas in or variations of previous ones until it seems like all the pieces of something more complete are there. This can be a bit trickier if there are time signature or tempo changes but it just takes a bit more time.

Things that seemed to be important factors for me:
  • Getting as much down for a demo as possible at once made for more consistent demos. A couple of them I just sat down at 1pm on a Sunday and worked for a 3-4 hours and got the full thing demo'd ready for presenting.
  • Making sure there were presets effects chains that could be dropped straight onto instruments to get something that sounded good right away, instead of getting caught up fiddling. There are many free plugins for guitars/drums out there that sound good so it doesn't really matter what you use, as long as you can just apply a 'distorted guitar' preset and be happy and carry on. It also means that when you've finished the demo it just sounds good already and doesn't need any real mixing before you share it.
  • Not second guessing things. It's easy to get stuck in "maybe it doesn't fit", etc. just run with what's there, you can always save another version. Get version 1 to done.
  • Having a Reaper template ready to go, with tracks for all the instruments with the correct VSTs already loaded and a click track etc. is the way to go here to maximise the time spent writing and not fiddling with software. 
With other commitments it took me a month to get all 5 demos down into a form where I'd shared them and we'd started talking about them. There were no vocals present at this point as there was still some indecision about what we would do on that front. In the future I'd get something down as a starting point for development.

Google Developer Group MK at Bletchley Park

The Milton Keynes GDG hosted their December meetup at The National Museum of Computing inside Bletchley Park. We had a detailed demons...