IndyWatch Education Feed Archiver

Go Back:30 Days | 7 Days | 2 Days | 1 Day

IndyWatch Education Feed Today.

Go Forward:1 Day | 2 Days | 7 Days | 30 Days

IndyWatch Education Feed was generated at Community Resources IndyWatch.

Monday, 18 June

23:03

How I built a handwriting recognizer and shipped it to the App Store "IndyWatch Feed Education"

From constructing a Convolutional Neural Network to deploying an OCR to iOS

The Motivation for the Project  

While I was learning how to create deep learning models for the MNIST dataset a few months ago, I ended up making an iOS app that recognized handwritten characters.

My friend Kaichi Momose was developing a Japanese language learning app, Nukon. He coincidentally wanted to have a similar feature in it. We then collaborated to build something more sophisticated than a digit recognizer: an OCR (Optical Character Recognition/Reader) for Japanese characters (Hiragana and Katakana).

Basic Hiragana & Katakana

During the development of Nukon, there was no API available for handwriting recognition in Japanese. We had no choice but to build our own OCR. The biggest benefit we got from building one from scratch was that ours works offline. Users can be deep in the mountains without the internet and still open up Nukon to maintain their daily routine of learning Japanese. We learned a lot throughout the process, but more importantly, we were thrilled to ship a better product for our users.

This article will break down the process of how we built a Japanese OCR for iOS apps. For those who would like to build one for other languages/symbols, feel free to customize it by changing the dataset.

Without further ado, lets take a look at what will be covered:

Part 1: Obtain the dataset and preprocess images
Part 2: Build & train the CNN (Convolutional Neural Network)
Part 3: Integrate the trained model into iOS

What the final app could look like (demo comes from Recogmize)

Obtain the dataset & Preprocess Images 

The dataset comes from the ETL Character Database, which contains nine sets of images of handwritten characters and symbols. Since we are going to build an OCR for Hiragana, ETL8 is the dataset we will use.

Images of handwritten produced by 160 writers (from ETL8)

To get the images from the database, we need some helper functions that read and...

22:15

These are the most effective microservice testing strategies, according to the experts "IndyWatch Feed Education"

Photo by Riccardo Annandale on Unsplash

Testing microservices is hard. More specifically, end-to-end testing is hard, and thats something well discuss in greater detail in this article.

But first, a quick story.

I learned just how hard microservice testing could be when I first dove into a tech stack with seven separate microservices. Each had its own code base, dependency management, feature branches, and database schemawhich also happened to have a unique set of migrations.

Talk about hectic.

The approach we took was to run everything locally. So that meant, at any given point in time when I want to run end-to-end tests, I would need to go through the following five steps for each of the seven microservices:

  1. Ensure I was on the correct code branch (either master or feature_xyz)
  2. Pull down the latest code for that branch
  3. Ensure all dependencies were up to date
  4. Run any new database migrations
  5. Start the service

This was just a baseline requirement to enable tests to be run. Often times, I would forget to perform one of these steps for a service and spend 1015 minutes debugging the issue. Once I finally had all the services happy and running, I could then begin to kick off the test suite. This experience sure made me miss the days of testing one big monolith.

So yes, I discovered that end-to-end microservice testing is hardand gets exponentially harder with each new service you introduce. But fear not, because there are ways to make testing microservices easier. Ive spoken to several CTOs about how they came up with their own creative ways of tackling this complex issue.

Common microservice testing methods

Unit Testing

A microservice may be smaller by definition, but with unit testing, you can go even more granular. A unit test focuses on the smallest part of a testable software to ascertain whether that component works as it should. Renowned software engineer, author, and international speaker Martin Fowler breaks unit testing down into two categories:

  1. Sociable unit testing: This unit testing method tests the behavior of modules by observing changes in their state.
  2. Solitary unit testing: This method focuses on the interactions and collaborat...

21:52

Creative Coding How to create a VJ engine in JavaScript "IndyWatch Feed Education"

Learn how to dynamically inject JavaScript into webpages

from radarboy3000 on Instagram

For years Ive been using the browser for my performances and installations using my own simple homegrown VJ engine. And now, after you learn a few simple tricks, you can too

A quick intro

Firstly, what is a VJ engine? you might ask. And maybe even: what is a VJ? Wikipedia defines the characteristics of VJing as:

The creation or manipulation of imagery in realtime through technological mediation and for an audience, in synchronization to music.

And a VJ engine is simply the software used for VJing.

But why would I build my own when there are so many VJ engines out there?

I never really loved VJ softwarethey always felt bloated and made everyones stuff look kinda the same. Its kind of like when you first got your hands on Photoshop, and just blended a bunch of stuff together, added some filters, and it was cool (because it was the 90s). But most of all, I wanted tighter and better integration between developing content and the sound input frequencies.

I rarely VJ these days, but it still drives most of my installations and performancesanywhere I need multiple visualisations I use RBVJ (the RB is for Radarboythats me) as a wrapper/player.

RBVJ has gone through a number of iterations over the years, as Ive bounced from Flash, to Processing, and finally now to JavaScript, all using the same simple system.

I had previously open sourced it and my content (before the days of Git and not really knowing there was a thing called open-source). It won a bunch of awards, and I saw my content being used in a whole bunch of places which was nice. So I thought it was time to make it available for others again, along with a bunch of content to show how I go about creative coding.

from radarboy3000 on Instagram

Ok, thats a long enough introduction. Show me the money, you say!

1. Structuring the Content

Essentially, a VJ engine is just a fancy content browser and player. So what we really need is a way to retrieve and play our content.

You could just dump all your content in a folder, but this system is what has worked best for me, allowing a simple structure for keyboard co...

21:31

JavaScript code cleanup: how you can refactor to use Classes "IndyWatch Feed Education"

Photo by Jennifer Burk on Unsplash

In smaller React projects, keeping all of your component methods in the components themselves works well. In medium-sized projects, you may find yourself wishing you could get those methods out of your components and into a helper. Here, Ill show you how to use a Class (instead of exporting individual functions and variables) to organize your code.

Note: I work in React so thats the example well discuss here.

Typical refactor

In a typical refactor, youd take a function on the component and move it to another helper.

From:

const MyComponent = () => {
const someFunction = () => 'Hey, I am text'
  return (

{someFunction()}

)
}

To:

import { someFunction } from 'functionHelper.js'
const MyComponent = () => {
return (

{someFunction()}

)
}

and

export const someFunction = () => 'Hey, I am text'

This example is really silly, but you see where were going:

  1. Take your functions and copy them over to a separate file
  2. Import them and call them as normal.

When things get complicated, though, youll have to pass in a bunch of stuff to those functionsobjects, functions for manipulating state, and so on. Today I ran into a problem where I wanted to extract three functions out of a component and they all required the same inputs (a resource and a function to update the resource). Theres got to be a better way

Refactoring with a class

I made a big demo for this post. You can see the code on Github. The initial commit shows all of the functionality inside the main component (App.js) and the subsequent commits refactor the code to use a class.

You can run this yourself and do whatever you want to it. Remember to yarn install.

We start with a component that fetches an object (mimicking the way we might do this from an API) with certain attributes on it: repeat (number of boxes), side (height and width), text, color. We then have a number of ways we manipulate the viewchanging the co...

21:17

An Intro to Web Scraping with lxml and Python "IndyWatch Feed Education"

Photo by Fabian Grohs on Unsplash

Why should you bother learning how to web scrape? If your job doesnt require you to learn it, then let me give you some motivation.

What if you want to create a website which curates the cheapest products from Amazon, Walmart, and a couple of other online stores? A lot of these online stores dont provide you with an easy way to access their information using an API. In the absence of an API, your only choice is to create a web scraper. This allows you to extract information from these websites automatically and makes that information easy to use.

Here is an example of a typical API response in JSON. This is the response from Reddit:

There are a lot of Python libraries out there which can help you with web scraping. There is lxml, BeautifulSoup, and a full-fledged framework called Scrapy.

Most of the tutorials discuss BeautifulSoup and Scrapy, so I decided to go with lxml in this post. I will teach you the basics of XPaths and how you can use them to extract data from an HTML document. I will take you through a couple of different examples so that you can quickly get up-to-speed with lxml and XPaths.

Getting the data

If you are a gamer, you will already know of (and likely love) this website. We will be trying to extract data from Steam. More specifically, we will be selecting from the popular new releases information.

I am converting this process into a two-part series. In this part, we will be creating a Python script which can extract the names of the games, the prices of the games, the different tags associated with each game and the target platforms. In the second part, we will turn this script into a Flask-based API and then host it on Heroku.

First of all, open up the popular new releases page on Steam and scroll down until you see the Popular New Releases tab. At this point, I usually open up Chrome developer tools and see which HTML tags contain the required data. I extensively use the element inspector tool (the butto...

18:46

The Ultimate Guide for Working with Side Projects "IndyWatch Feed Education"

Over the last ten years or so, Ive made a habit of maintaining at least one side project at any given moment.

Those side projects have served a couple of purposes:

  1. Learn new things: Trying out a new technology, a new field, or learning a new skill.
  2. Build something: Its nice and fulfilling when you can show your product to others, or take a look at something and know youre the one that created it.
  3. Refreshing: Its a great way to get out of your comfort zone and do something different from your day job.

Ive listed these reasons in order of importance to me. Personally, most of the times my primary goal was to learn something new, but this can be different for you.

There are a lot of articles about why side projects are important. You can read about them here or here.

Here are some of my the side projects Ive worked with:

  • 3D house model: Building a 3D model of my house back in the day. I even thought about applying for a job at Pixar. So to increase my chances of getting a job as a software engineer there, I decided to learn how to work with Maya.
  • Bill splitting app: I built a bill splitting app while exploring new fields I wanted to experience, like developing a user interface product. Thats why I created the Bill Splitter - Tip Calculator app, which later helped me get hired by a mobile startup.
  • Decorate my childrens room: When I was young I loved drawing, but I never had the time to do that when I grew up. So, I used the opportunity to learn how to draw with acrylic colors and sketch some stuff in our childrens room before my older daughter came into the world. Later, I did the same for our baby boy.
  • Write a blog: To practice my English, I started to publish content regularly. I might have written stuff that went viral, but my English is still not as good as I wish it was.

Those are just a few representative examples to give you a sense of what a side project can look like and what you can achieve.

...

18:28

How to tune up your Progressive Web App to get a perfect audit score "IndyWatch Feed Education"

Progressive Web Apps (PWAs) quickly became the hottest development platform during the last year. Lets take a look at what you need to do to adhere to the PWA standards.

Articles about the PWA concept are all over the place. I will focus on the actual steps that need to be done to have the PWA fully aligned to the specification. I will provide a GitHub link with the list of changes for each step I performed to allow you to easily try it yourself.

Prerequisites

I will start with that simple Angular application I used for showcasing the combination of Angular and PWA approach in my previous article. I have upgraded it to Angular v6 and Kentico Cloud SDK v4.

Upgrade changes

This application is a simple listing of interesting places stored in a headless CMS loaded by the SDK. The app already has these two perks that make it a PWA app:

  • manifest.json with a set of icons used when the app is installed in the system.
  • service worker implementation used for caching of the application skeleton (called the app shell) and the data from the headless CMS.

Although the app is ready to be installed and used, it still needs a few touches to meet the PWA specification.

How to get through the PWA checklist

To verify whether the app meets all the criteria defined by the Google checklist, one can use various tools these days. The most popular is called Lighthouse.

Google checklist

Lighthouse is already embedded into the Google Chrome browser developer tools on the...

18:20

Web Caching Explained by Buying Milk at the Supermarket "IndyWatch Feed Education"

If you have ever bought milk at the supermarket, then you can understand server-side and browser-side caching.

If you are an avid internet user (you probably are), you have benefitted from caching over and over again. But, you might not know when or how it is working its magic behind the scenes.

From a developers perspective, caching makes it much easier to build high-performing web apps and web servers. Instead of needing to constantly optimize servers that are overwhelmed by thousands of requests, developers can implement caching protocols to make life much easier.

Since caching might make a difference between 1 second to load a page and 2 seconds, the impact can feel a little bitunderwhelming. But, its necessary if you want to handle a high volume of users.

After using caching in a past web app, I realized that there had to be a better way to explain it than just walking through the terminology. I noticed that it aligned very well with the pathway of milk from a farm to your refrigerator, so I figured that would be a better way to explain it.

In order to understand this guide, you just need to know about the basics of web servers. Lets get into it!

What would the internet look like without caching?

Before we get into caching, lets think about what the internet would look like without caching. Imagine, for a moment, that you are living in the 1700s or 1800s in a rural area. You own a farm, and there is no refrigeration available. You have a few cows on your farm, but their milk is not nearly as valuable since it will spoil quickly.

Quick interruption: Some cultures still do not have access to refrigeration. They will either drink raw milk directly from a cows udder, or mix milk with grains and let it ferment. Interesting.

ANYWAYS, you want to sell your milk to others in your village. But, they will have a very limited time to drink the milk. Lets say that one of your cows can produce a gallon of milk per day. But, if too many people show up to your farm looking for milk, you will need to send some home and ask them to wait til the next day.

Also, you cant even think about adding more cows and scaling up your operation since you have such limited distribution. Only the other members of your village can buy your milk. You have some clear limits.

...

16:36

Wanna be a developer? Here is what you need to take into account. "IndyWatch Feed Education"

It might not be as easy as youve thought

A common myth about software developers is that theyre boring and dry geeks, who were math geniuses at school, that spend hours in front of computer screens writing code.

Yes, developers may spend their life in front of computer screens writing code. However, there is much more to it than just coding like a machine the whole day. For me personally, being a developer means you have the chance to build cool stuff by yourself.

I started self-learning front-end development my sophomore year. My journey has been filled with self-doubt and hurdles, along with joy and extreme excitement. I never imagined I could experience all that while learning something.

Sometimes, even now, I still wonder whether I am following the right path. However, by telling myself that there is nothing more pleasurable than being able to do what I love, I keep putting my head down and I continue working.

It has been around two years since I started writing my first lines of code. After many hours of practicing, with sometimes feeling like giving up followed by temporary discontinuity, I would like to share with you a few things that I wish I knew from the beginning.

Dont treat coding as a leisure interest

First and foremost, if you think that you need to be serious while coding, youre 100 percent right. It is definitely true that you will probably not spend time doing something you dont like.

However, doing it just on a hobby basisthat is, you only code when you feel like doing it without any specific commitment or schedulewill probably not lead you to the career that you have always wanted.

In addition, when you encounter obstacles and difficulties, are you sure that you will have enough patience to keep your little interest alive? Rather, you might end up giving up or potentially going through a long stagnation in the middle.

Therefore, you should be incredibly dedicated to your passion, my friends. Yes, I am sure that all of us developers have a great interest in coding and technology in general.

However, passion is nothing without the right execution. By committing to a specific goal along with an appropriate schedule, you are building milestones along your journey. Put in a huge commitment to your given timeframe. Specify which skills and technologies you want to learn over a certain period of time. Then youll be closer than ever to making learning how to code an imperative part of your life.

Figure out which technologies you need to focus on

Once youve started taking coding seriously, the next step is to be honest with yours...

Thursday, 10 November

02:43

Protected: Anne Zahalka captures our past and present "IndyWatch Feed Education"

This content is password protected. To view it please enter your password below:

IndyWatch Education Feed Archiver

Go Back:30 Days | 7 Days | 2 Days | 1 Day

IndyWatch Education Feed Today.

Go Forward:1 Day | 2 Days | 7 Days | 30 Days

IndyWatch Education Feed was generated at Community Resources IndyWatch.

Resource generated at IndyWatch using aliasfeed and rawdog