I'm very particular about compile-time checks and runtime safety when it comes to writing software, and it becomes apparent if you read my other posts. In 2018, when I first came across Ethereum, also known as the 'world computer,' I knew nothing about the technologies behind it except that I'd need to use Solidity for this computer. The Solidity language website v2.2.0 stated, "Solidity is a high-level language whose syntax is similar to that of JavaScript, and it is designed to compile to code for the Ethereum Virtual Machine."

The reference to JavaScript alarmed me. How come someone use a potentially risky language like JavaScript to manage money? I know it mentioned only the syntax, but the reference was strong enough to discourage further exploration in that area, given at that time it didn't look something worth attention.

However, it turned out to be the complete opposite: Solidity is a type-safe, runtime-safe language that only shares a C-like syntax with JavaScript. Sadly, I discovered this three years later.

I was going to provide some examples here, but I quickly realized that it makes no sense due to the massive differences at the conceptual level. Things like number overflow, limited or no string operations, explicit number casting, manual data location specification, limited local memory, and fixed-sized arrays—none of these are applicable to JavaScript.

Seriously, these things only make sense in Solidity (though not always a good idea):

• Thoughtfully design the memory layout when inheriting a class for your delegate implementation.
• Change the order of variables to align them in words and reduce the cost of reading and writing operations to storage.
• Pack array indexes into u256 using binary operations to save on gas.
• Check variable values to avoid overflow and underflow.

If anything, to me, Solidity is very close to C-- with the memory management in terms of syntax and proximity to the "hardware" but couldn't be further from JavaScript.

Service diagram

Essentially, the service consist number of the components:

• Terms Database - dataset of the words
• Voice generator - engine which generates human-alike voices
• Picture lookup service - huge part which is responsible for finding relevant background picture
• Video generator - renderer of the video which composes the clips and voices
• Youtube Uploader - implementation of the client Youtube API
• Management panel - very basic web-based admin panel allowing to observe current status, database and statictics
• Statistics extractor - a regular fetching some statistics data

The central focus of the service is a Term - one or few words which need to be pronounced. Service is built around the pipeline which takes the term and transforms it into the uploaded Youtube video.

Bear in mind this project is at the MVP stage. That means there are several compromises I have chosen to reduce Time To Market. It's never late to improve code if the project is useful.

Technologies used

I could say it uses most-of-the-buzzwords such as Computer Vision, Mesh API, Cloud Services, Docker, Microservices and it won't be a lie. But we always want to see more specifics:

Generator service:

• Kotlin - of course, there are no other candidates. Most of the engine is built on it
• http4k - webserver written in kotlin, for kotlin
• kotlinx-html - dsl for html. There is no javascript in the project as such
• kmongo - as a DAO layer for the db
• java-pixabay - for Pixabay API access, the original project has been abandoned, I had to fork it

Video generator and render

It's a microservice working in the separate container, communicating with the engine via HTTP

• python3 - because of moviepy
• moviepy - the best programmatic video generator; I found anything similar for JVM.
• imageio - for image manipulation
• cherrypy - apparently the easiest way to expose python function via http REST service

Infrastructure:

• jib - jvm docker image generator
• mongo - default database for experiments/MVP
• docker - as a runtime
• docker-compose - for container orchestration
• docker-machine - for provisioning
• make - as a frontend for the deployment commands

Cloud API and services:

• AWS Rekognition - cloud computer vision API. It allows to filterout pictures with faces(often they create a lot of noise) and image image labeling for choosing the best picture
• AWS Polly - TTS engine provided by Amazon. Previously I tried one from MS Azure but quality wasn't satisfying. Polly generates nearly perfect voice with different accents
• YouTube API - for video uploads and statistics collection
• AWS EC2 - for hosting
• PaperTrail - for logs from the docker containers

Challanges

Well, there were a lot of issues, mostly related to the external services

1) YouTube API has limitations - each call counts towards daily quota(which is 1M units). From my experience, it's only possible to upload about 50 videos a day. Although I'd like to upload way more than limit set that didn't bother me much since the process is automated
2) moviepy heavily leaks memory. From my experiments that after 10 rendered videos python process held about 2Gb of RAM. Since it's MVP I have chosen the simplest solution - just restart microservice. More precisely, to configure docker-swarm to kill it once it consumed too much memory. I believe it's a very practical decision for the given project stage.
3) To make the video stand out from others it has to have a relevant background picture for the term. If the user looks for the "how to pronounce tomato" it's more likely that video with tomato on the background would be chosen rather than one with grey colour. To find images I used Pixabay API(if you like service don't forget to donate them too!). For the obvious reasons often some irrelevant pictures are returned, so I had to filter irrelevant pictures using Amazon computer vision.
4) Imagemagik policies hurt. It's a great library but I found it tricky to configure since it has a configuration file where defaults are very tight. For example it's impossible to generate video into the /tmp folder by default. Thanks to docker it's very easy to build up the image with embedded configuration.
5) Apparently, docker-compose has changed the behaviour for the container limits so I had to downgrade configuration file from version 3.3 to 2.3
6) I wanted to keep MongoDB outside of the container on the host machine for my personal reasons. If you ever tried to do so you know it's not easy. The container ecosystem is pushing a user to use containers only. I ended up binding /var/lib/mongodb/mongod.sock from host to container and use jnr-unixsocket to make mongo to use unix socket instead of TCP
7) Youtube API documentation seems to be very convoluted, I had a hard time to understand how to go from the simple youtube upload to something like "create a playlist if need and then specify description along with tags and location of the video in different languages"

Enjoyable parts

This project is actually quite interesting to work on. It uses many external APIs, works with computer vision(a lot of fun with debugging!), etc

• kotlin is soo nice, as usual. Can't imagine myself using python which can expode after every single type or java where I would write a few books and still it's not that clean
• Writing web pages in kotlin with kotlinx-html is really fun. Just think - statically typed html templates!
• Amazon Rekognition works like a magic, I'd say in 90% it sees what I'd say about the picture. Prices are very competitive for my use case
• Sealed classes work really well for the statistics collection and voices description
• kmongo allows to express db queries via staticly typed DSL. As most ORM it fails on the complex constructions but perfomance of the DB communication is never consern for this project
• java-pixabay library has been outdated, I made a few PRs but author had not got back to me. For that reason I continued to work on my fork - ruXlab/pixabay-java-api

Code snippets

I'd like to highlight some code used in PronounceMe service

kotlinx-html templates

private inline fun <reified T : Any> BODY.dumpListAsTable(
    list: List<T>, fields: Collection<KProperty1<T, *>> = T::class.memberProperties
) = table("table table-striped table-hover") {
    thead {
        tr {
            for (field in fields)
                th { +field.name }
        }
    }
    tbody {
        for (row in list) {
            tr {
                for (field in fields)
                    td { +field.get(row).toString() }
            }
        }
    }
}
private fun youtubePlaylists(req: Request): Response = pageTemplate("Youtube playlists", autoreload = false) {
    dumpListAsTable(
        youtubeClient.getPlaylists(),
        listOf(YoutubePlaylist::id, YoutubePlaylist::title, YoutubePlaylist::itemsCount, YoutubePlaylist::description)
    )
        . . . .
}

Outputs:

Navbar and html body

body {
    nav("navbar navbar-expand-md navbar-dark bg-dark fixed-top") {
        div("collapse navbar-collapse") {
            ul("navbar-nav mr-auto") {
                WebApp.webRoutes.filter { it.verb == Method.GET }.forEach {
                    li("nav-item") {
                        a(it.url, classes = "nav-link") {
                            +it.description
                        }
                    }
                }
            }
            ul("navbar-nav mr-auto") {
                li("nav-item") {
                    if (PronounceApp.isRunning.get()) {
                        h2 {
                            span("badge badge-danger badge-secondary") { +"Generator is running" }
                        }
                    } else {
                        form(action = "/forcestart", method = FormMethod.post, classes = "form-inline") {
                            button(classes = "btn btn-success", type = ButtonType.submit) { +"Force start" }
                        }
                    }
                }
                li("nav-item") {
                    a("https://papertrailapp.com/groups/XXXXXXX/events", "_blank", "nav-link button") {
                        +"logs"
                    }
                }
            }
        }
    }
    main {
        h1 { +title }
        builder(this@body)
    }
}

Result looks like:

Decent design for the private admin panel to be used every once in a few months by a single person, isn't it? :)

Server and routes

Handlers are defined as a list of routes with URL and handler function as expected in http4k

val webRoutes = listOf(
    Route(GET, "/ping", "Ping it") { req -> Response(OK).body("pong") },
    Route(GET, "/stat", "Some stats", this::stat),
    Route(GET, "/events", "Recent events", this::recentEvents),
    Route(GET, "/", "All urls available", this::root),
    Route(GET, "/config", "Runtime config", this::config),
    Route(GET, "/stat_channel", "Channel statistics", this::channelStat),
    Route(GET, "/youtube_playlist", "YT playlists", this::youtubePlaylists),

    Route(POST, "/forcestart", "Force start", this::forceStart),
    Route(POST, "/createplaylist", "Create playlist", this::youtubePlaylistCreate)
)

The webserver itself is literally 3 lines of code

routes(*webRoutes.map { it.url bind it.verb to it.handler }.toTypedArray())
    .asServer(SunHttp(port))
    .start()

The heart of the image lookup component:

fun findImageForWordWithCandidates(
    word: String,
    category: Category?,
    stopList: List<String>,
    mandatoryList: List<String>? = null,
    allowFaces: Boolean = false,
    pixabyPage: Int = 1
): ImagesWithCandidates? {
    val stopList = stopList.mapTo(HashSet(), String::toLowerCase)
    val mandatoryList = mandatoryList?.mapTo(HashSet(), String::toLowerCase)
    val allImages = imageLookup.searchPixabay(word, category, pixabyPage)
        ?.shuffled()
        ?.also { log.info("findImageForWord: got {} images for {}", it.size, word) }
        ?.mapNotNull {
            // fetch pictures locally
            runCatching {
                ImageRuntimeInfo(it.largeImageURL, URL(it.largeImageURL).asCachedFile("pixaby-${it.id}-large"), pixaby = it)
            }
            .onFailure { log.warn("findImageForWord: during image saving", it) }
            .getOrNull()
        }
    val images = allImages
        // filter pics with faces if necessary
        ?.let { if (allowFaces) it else withoutFaces(word, it) }
        ?.also { log.info("findImageForWord: got ${it.size} pics after face filtering") }
        // image labelling
        ?.let { findLabels(word, it) }
        // exclude stop list words
        ?.filterNot { it.normalizedLabels.any { it in stopList } }
        // exclude images without mandatory words
        ?.filter {
            if (mandatoryList == null) true
            else it.normalizedLabels.any { it in mandatoryList }
        }
        ?.also { log.info("findImageForWord: got ${it.size} pics after filtering by label") }
        ?: return null

    if (images.isEmpty()) {
        log.warn("findImageForWord: No eligible images were found for {}", word)
        return null
    }

    val sortedImagesByConfidence = images
        .map {
            // find the best matches by original word
            val labelWithWord = it.labels
                .sortedByDescending { it.confidence }
                .firstOrNull { it.name.contains(word, ignoreCase = true) }
            it to (labelWithWord?.confidence ?: -1.0F)
        }

    log.debug("findImageForWord: ${sortedImagesByConfidence.size} candidates for $word: \n{}",
        sortedImagesByConfidence.joinToString("\n") { "   - ${it.first} with ${it.second} confidence" })

    val firstBestMatch = sortedImagesByConfidence
        .firstOrNull { it.second > 0.0F } // return first by confidence
        ?.first

    log.info("findImageForWord: best match for {} by label in word - {}",
        word, firstBestMatch)

    if (firstBestMatch != null)
        return ImagesWithCandidates(firstBestMatch, allImages)

    // we don't have best extact match by word in labels
    val firstMatchByConfidence = sortedImagesByConfidence.firstOrNull()
    log.info("findImageForWord: good match by confidence for {} - {}",
        word, firstMatchByConfidence)

    return ImagesWithCandidates(firstMatchByConfidence?.first, allImages)
}

Clips compose

Pardon me for my python

for idx, _ in enumerate(voice_title_clips):
    prevoice_clip = CompositeVideoClip([static, voice_title_clips[idx]], size=screensize)
    prevoice_clip.duration = pre_voice_pause
    postvoice_clip = prevoice_clip.set_duration(post_voice_pause)
    voice_title_clips[idx] = CompositeVideoClip([static, voice_title_clips[idx]], size=screensize)
    voice_title_clips[idx].duration = voice_clips[idx].duration * voice_repeats + voice_repeats_pause_times * voice_clips[idx].duration
    silence_clip = silence.set_duration(voice_clips[idx].duration * voice_repeats_pause_times)
    voice_title_clips[idx].audio = concatenate_audioclips(intersperse([voice_clips[idx]] * voice_repeats, silence_clip))
    clips = [prevoice_clip, voice_title_clips[idx], postvoice_clip, static.set_duration(pause_between)]
    voice_title_clips[idx] = concatenate_videoclips(clips, padding=-1, method="compose")

What is next

Subscribe for the blog to see where this project goes. Breaking news is awaiting!

Checkout more project updates from posts grouped by #pronounceMe hashtag

General convention

Although one might think that kotlin has inherited all the base collection types from the Java it's not quite true. Kotlin transparently maps existing Java collections into the Kotlin by using some tricks such as typealiasing. Collections hierarchy in Kotlin make code even more safer by imposing separation between mutable and immutable data structures. Take a look on the interfaces diagram:

Having dedicated interfaces for immutable collections makes expressions are purely functional - no need to worry if api consumer modifies list on the way or even worse, attempt to insert into the immutable collection(goodbye UnsupportedOperationException!). Indeed, immutability is enforced in compile time by contract.

A note about Iterable vs Sequence

Those are very similar types of the base entities even with the same signatures, let's take a look

public interface Sequence<out T> {
    public operator fun iterator(): Iterator<T>
}

public interface Iterable<out T> {
    public operator fun iterator(): Iterator<T>
}

Those two base classes define the way data will be processed in the chain of the calls:

• Operations on Iterable produce result immideatelly, so the full intermideate result will be passed between calls in the chain. The result is evaluated eagerly after each step.
• Operations on the Sequence treat data items comming thorough as it would be an infinite stream, the closest analogy would be java8 Stream or RxObservable. Items passed via the chain of the calls one by one. Result is evaluated lazily.

As for now we focus on the Iterable and it's descendants(Collection, List, Map, etc..) . Luckily, many operations exist for both interfaces with exactly the same signatures

Simple list transformations filter, map, forEach

Those are the probably the most widely used operators and they do exactly after their name. The provided function is applied to the each element of the operation

val adminNames = users
  .filter { it.isAdmin }
  .map { it.name }

pupils.forEach { 
  println("${it.name}: ${it.score}")
}

filter* and map* families

There are way more similar operations provided in the Kotlin stdlib giving extra flexibility when it need:

val userList = users
  .filterNotNul()
  .filterNot { it.isBanned }
  .mapTo(mutableHashSet()) { it.userId }
  .mapIndexed { (idx, userId) -> "#${idx}: {it.userId}" }

In many occasions you'll find the same pattern - verb [not] [indexed] [to]. No need to memorise - the names come out intuitively:

Operations returning single element: first, last, single, elementAt, get

first and last return first and last elements (obviously).

val firstUser = users.first()
val firstAdminUser = users.first { it.isAdmin }
val lastBannedUser = users.last { it.isBanned }

single returns one element and throws exception if more than 1 element in collection matches the predicate

val oneLove = listOf("java", "kotlin",  "javascript").singleOrNull { it == "kotlin" } 

Also those operations can have return alternative value - provided by closure or null:

val oneLove = languages.singleOrNull { it == "kotlin" }
val tenthWinnerName = user.getOrElse(10) { "NO WINNER" }
val secondPerson = user.getOrNull(2)

Aggregation operations count, average, min, max

Again, intuitively those operations perform aggregations:

val avgScore = pupils.average { it.score }
val topStudent = pupils.max { it.score }
val channagingStudent = pupils.min { it.score }

Conditional oprations all, none, count, any

val numberOfTopStudents = pupils.count { it.score > 4.5 }
val allPassed = pupils.all { it.score > 2.0 }
val hasNeedleInHaystack = heap.any { it.object == NEEDLE }
val allGood = results.none { it.error != null }

List to Map transformation associate*, groupBy

Both operations produce a Map and they are different on how keys are collided. While assciate* simply overwrites existing value with associated key, groupBy adds value to the list of values:

val usersById = users.associate { it.id to it } // result type: Map<UserId, User>
val usersById = users.associateBy { it.id } // same output
val pupilsByScore = pupils.groupBy { it.score } // result type Map<Int, List<Pupil>>

Many more

There a way more functional operations over collections are available in Kotlin stdlib such as fold, reduce, minus(-), plus(+), contains(in) etc:

// result - list of the both users
val allUsers = fbUsers + twitterUsers 

// result - elements of allUserIds which are not in bannedUsersIds
val activeUserIds = allUserIds - bannedUsersIds 

// result - the longest length of the name
val longestName = names.reduce { longest, item -> if (longest.length < item.length) item else longest }

// result - same as above, the longest length of the name
val longestLength = names.map(String::length).fold(0, ::max))

// result - if Wally was there
val isWallyLovesKotlin = "Wally" in kotlinLovers

Those extension functions are very intuitive and widely used, essentially can cover most of the everyday tasks.

Conclusion

Kotlin collection functions provide a lot of flexibility to express your ideas and business logic in very concise, clear and functional way

Hopefully you found this article useful for you, please check out other posts by #kotlin-showoff hashtag

Thanks to my wife, I recently learnt London Bus Stop skill:

• still in the top 30 skills in the area because I'm receiving $100 credit for AWS every month;
• it's not listed anymore! That fact slipped through the cracks!

Statistics

I've noticed that only 20 unique users are using this skill every month but didn't take attention for that. I wrongly assumed it's related to the low popularity and/or presence of the competitive skills. I was clearly wrong. Number of Londoners who have Alexa is only growing, so audience of the skill should.

Clearly no one has enabled skill, even by mistake. It never made me doubt.

The issue

Apparently, Amazon imposed new requirements for the skills and performed some sort of automated testing. It seems that problem was with session finishing - skill should be reacting on the default "Quit" intent but it didn't, at least in all required cases.

I didn't receive any notifications or simply ignored them - Amazon sends bunch of the promo emails almost every week regarding new features in Alexa API and toolkit around it. Disappointing.

So skill was active for people who already has activated it but new audience can't enabled it anymore. 10 mins fix and Voilá - fixed. Automated review passed in one minute + 4 days for the review by human and skill went back to the listing.

5 days statistics since it back

Skill started to get new audience!

Conclusion

Well, monitor emails? To be honest I can't think of the generic automated system which will check presence in the store listing. This monitoring has to be thought and implemented through parses I suppose? Alternatively, data can be collected from the dashboard and analysed for the down trend.

Will see how it goes, I'll post update some time in the future. Stay tuned.

In meanwhile, if you're in London please checkout the skill and rate should you like it.

Alexa, open bus stop

For folks who got lost on the word "extension functions" - it's a way to attach a function or property to the instances of the existing classes. For example, val d = 10.twice()It's very much like a classic Java Util classes with method twice(int) but done in a very clean way. Visually it looks like you're calling a member of the class, but in reality, the compiler calls your function passing receiver as an argument.

T.let

let allows passing given reciever into the parameter of the lambda, returns result from it. Essentially it's an map operator on the single value

val imageData = db.getUserById(userId)
    ?.let { it.getProfilePicture().url }
    ?.let { fetchData(it) }

How to use?

It's very useful when it comes to the conditional on nullable object, think of functional replacement for if (user != null) user.ban():

user?.let { it.ban() }
// or even
user?.let(::ban)

Often we use expressions in the string interpolation, .let can be very handy:

println("Hello ${db.getUser(id)?.let { it.name } ?: "NONAME"}")

T.apply

apply calls given lambda in the context of receiver, similar to the with found in JavaScript and Groovy.

User().apply {
    name = "user name"
    password = "qwerty"
}

How to use?

I found it's a way to patch encapsulate configuration/finish object initialisation. It helps to perform number of operations within lambda block which are logically related to the operation such as factory initialisation or object creation. apply returns the receiver object(object it was called on)

So instead of

val awsS3Client = AwsS3Client()
val credentials = AwsCredentialsFactory("key", "secret")
credentials.setForceHttps(true)
awsS3Client.credentials = credentials
awsS3Client.bucket = "my_bucket"

We can have nicely organized initialization block:

val awsS3Client = AwsS3Client().apply {
    credentials = AwsCredentialsFactory("key", "secret").apply {
        setForceHttps(true)
    }
    bucket = "my_bucket"
}

Nice and clean, without messy noise and scope pollution!

T.also

It behaves exactly as .let with one exception - with return result. .also takes receiver and passes it as an argument for the given lambda, returning the same receiver

val bannedUser = db.getUserById(userId)
   .also { println("Got user $it") }
   .ban()

How to use?

From my experience it can be used for debugging - as demonstrated above. Also, it can be used a replacement for the .apply, one can say it's more clear - based on the example for .apply:

val awsS3Client = AwsS3Client().also {
    it.credentials = AwsCredentialsFactory("key", "secret").apply {
        it.setForceHttps(true)
    }
    it.bucket = "my_bucket"
}

T.run / with()

These have very similar behaviour - they both change the context of the function but unlike .apply the result of the function would be something lambda returned

val userId = User().run {
    name = "username"
    password = "foobar"
    db.save(this).id
}

val userId = with(User()) {
    name = "username"
    password = "foobar"
    db.save(this).id
}

These two have identical behaviour - in both cases object is initialised, saved and generated id is returned - the result of the last line expression

How to use?

Well, I didn't find it used often in my code, but the example above may give you some idea

Combo

Some examples from the real projects

// filter out images with faces, print debug info, annotate 
// image with labels and filter images by stop words
val images = allImages
    // filter pics with faces if necessary
    ?.let { if (allowFaces) it else withoutFaces(word, it) }
    ?.also { log.info("findImageForWord: got ${it.size} pics after face filtering") }
    // image labelling
    ?.let { findLabels(word, it) }
    // exclude stop list words
    ?.filterNot { it.normalizedLabels.any { it in stopList } }


// Prepare video upload request for the YouTube API:
val video = Video().also {
    it.snippet = VideoSnippet().also {
        it.set("categoryId", "27")
        it.set("description", description)
        it.set("title", title)
        it.set("tags", tags)
    }
    it.status = status
}

I hope you found this article useful for you. Check out other posts about functional constructions to get the most out of kotlin

Essentially, elvis operator lvalue ?: rexpression is returning left value if it's not null or executes rexpression otherwise. The crazy thing about kotlin is most of the constructions are expressions and that gives another way to express business logic.

Returning alternative value

In the simplest case we can return alternative value if left value is null, for example

val userList = cache.get(userId) ?: fetch(userId)

Literally means: get value of cache.get(userId) if it's not null or fetch(userId)

In Java it can be expressed as

List<User> userList = cache.get(userId);
if (userList == null) userList = fetch(userId);

Also, it's possible to chain elvis operators

val userList = memoryCache.get(userId) 
     ?: diskCache.get(userId)
     ?: fetch(userId)

Looks lovely and clean, isn't it?

Early return with elvis

fun displayUsers() {
    val userList = cache.get(userId) 
        ?: return
    updateUiWithNewUserList(userList)
}

In this case return if userList is null function will return immideatelly

Throw exception with elvis operator

In Java code we often need to manage an exceptional situation and allow callers to handle it themselves. Take a look on this example:

void displayUsers() {
    List<User> userList = cache.get(userId);
    if (userList == null) 
        throw new NoDataAvailableException()
    . . . 
}

Employing elvis operator in kotlin it looks more concise and logic is encapsulated within one expression:

fun displayUsers() {
    val userList = cache.get(userId) 
        ?: throw NoDataAvailableException()
    . . .
}

Looks cleaner, isn't it?

Elvis operator with in loops

In the same way as we used early return it's possible to make elvis to work with break and continue, in case if you're still using loops

fun top10Score(users: List<User>): Int {
    var totalScore = 0
    for(i in 0 until Math.min(users.size, 10)) {
        val score = users[i].gameProgress?.score 
            ?: continue
        totalScore += score
    }
    return totalScore
}

If either gameProgress or score are null loop will continue to the next element.
By the way, there is a way expressive functional way to implement the same logic:

fun top10Score(users: List<User>): Int = users
    .take(10)
    .fold(0) { acc, it -> acc + (it.gameProgress?.score ?: 0) }

It works not only for nullable types

Kotlin has numerous extension functions allowing to express flow in the concise and functional manner. We'll dive into this topic in the following articles.

But here is a spoiler for you, say it's need to throw exception in validator when user didn't provide name or it was empty. From data point of view null isn't same as empty string, empty string isn't same as a string full of spaces. But in our business logic it's practically the same since no name is provided. Here is an example of handling this situation:

val name = user.name?.takeIf(String::isNotBlank)
    ?: throw UserNameIsEmptyException()

takeIf returns receiver object if predicate is true. Even user.name is not null but consist of empty string takeIf will return null because it doesn't satisfy condition. Exception will be thrown in this case and it's guaranteed by compiler that name is not null

Conclusion

Elvis operator in kotlin is very powerful and helps to handle conditional logic naturally within the data flow

This is a first of this series of posts unioned by tag #kotlin-showoff

As from my experience it takes literally two days for average developer to switch to kotlin from java and start to write meaningful business logic

As time goes by code can use more kotlin specific constructions which make code much more readable and expressive. In this blog post I'd like to show off some of the nice functional constructions which help to make code more idiomatic.

Extracting data from the nullable types

Say, we have an class Address within UserInfo which holds information about the given user's address. UserInfo is a part of the Person. UserInfo can be nullalbe as well as Address but we want to extract city name. In java it would look like:

    Person person;
    String city;
    if (person.userInfo != null && person.userInfo.address != null && person.userInfo.address.city != null)
        city = person.userInfo.address.city;
    else city = "Unknown city";

    System.out.println("User is in " + city);

If you think it's a safe code I have a bad news for you. Savvy java developer can immediately spot the issue - between checks and assigment to the city whole bunch of things can happen on another threads and eventually we can end up with accessing null. Hello NPE!

The correct approach would be to capture values while we're performing the check:

    UserInfo userInfo = person.userInfo;
    if (userInfo != null) {
        Address address = userInfo.address;
        if (address != null) {
            city = address.city;
        }
    }
    if (city == null) 
        city = "Unknown city";

    System.out.println("User is in " + city);

Expressivity is not Java best strength. But wait, with Java 8 we can make it a bit more readable (and still safe):

    city = Optional.ofNullable(person.userInfo)
            .map(userInfo -> userInfo.address)
            .map(address -> address.city)
            .orElse("Unknown city");

    System.out.println("User is in " + city);

Looks better, isn't it? Wait for kotlin

println("User is in ${person.userInfo?.address?.city ?: "Unknown city"}")

Sorry java ¯\_(ツ)_/¯

So what exactly happens here? The chain person.userInfo?.address?.city returns either city if it's not null or null. Then using elvis operator we make whole expression to return extracted city value or string "Unknown city". Even more, we can concatenate everything in one string as an expression

You can eat your cake, and have it too

Is it safe one might wonder? Yes, it is. Look into the generated code:

public final class KotlinImplKt {
   public static final void kotlinImpl() {
      StringBuilder var10000;
      String var4;
      label14: {
         Person person = new Person();
         var10000 = (new StringBuilder()).append("User is in ");
         UserInfo var10001 = person.userInfo;
         if (var10001 != null) {
            Address var3 = var10001.address;
            if (var3 != null) {
               var4 = var3.city;
               if (var4 != null) {
                  break label14;
               }
            }
         }

         var4 = "Unknown city";
      }

      String var1 = var10000.append(var4).toString();
      boolean var2 = false;
      System.out.println(var1);
   }
}

Check out other posts in #kotlin-showoff

The demo project I prepared is a web site allowing user to upload pictures into GCP Storage, automatically annotate content using Vision API, synthesise voice which describes content of the image.

Web service is using kotlin, http4k, kotlinx.html, GCP APIs and has no JavaScript, which is very rare nowadays

The presentation is available here

GitHub repository with project: https://github.com/ruXlab/kotlin-gcp-vision-tts-demo

But things went really wrong, demo daemons have attacked my computer(yes, it's a silly excuse!)

What went wrong

Normally, before presentation I plug my laptop to the projector and everything works well, but not this time. Here is a postmortem of my effort to run presentation

• Projector was recognised by my laptop and picture was shown for the 5 seconds. After than it started to glitching and blinking like a disco lights.
• I was given HDMI to mDP converted which didn't help at all
• I started to panic because it's the only laptop I have. Luckily, person from Google Campus gave me the MacBook which works perfectly. But it had a tiny issue - no development software was installed there, even no java VM. Also, I'm terrible with macs
• I have tried to reboot system many times trying different kernels. Because of that I lost all open windows and prepared environment.
• I realised that I just need to stream my screen to Mac and it would be a solution. I just need to broadcast screen to local network
• Not that easy, huh. Google Campus WiFi is well protected - it doesn't allow members to connect to each other. Hence, there is no way Mac can consume image from my laptop through local network
• That situation absolutely derailed me since it was time to start meetup and audience was already in the room.
• Bingo! I realised I can use my phone as a router to allow laptops to talk to each other over LAN. It sounded good to me!
• Not that easy. There was no 4G coverage, no signal at all inside the building(it's a well known Google Campus issue). That means no cloud APIs are available :(. So I had to run to the window searching for the cell network signal
• After all I was trying to google simple recipe for vlc I failed, and the time pressure was very high.
• Deep breath, step back, "How do I deliver screen picture over network?". Google Hangouts, of course!
• "All sorted" - I thought.
• Nearly sweating and thirsty I have started presentation. Once I finished my slides I switched to Idea and quickly prepared environment for demo(which I lost during reboots).
• Idea started to play badly. Apparently it didn't replace title of the project so I was thinking that I was editing original project(with all code completed) instead of demo. I didn't realized that from the beginning so after few lines of code I had to open another "demo" folder which apparently was already opened.
• Idea didn't allow me to do so and I renamed folder "demo" to "demo1", continued to write code. At some point Idea stopped running application even "run" icon has gone from the "main" method. Disaster.
• 2 mins of silent voodoo in the command line in front of the dozens people and I still did not have a clue.
• My time was running up and organisers proposed me to continue after next talk. Unfortunately continuation never happened since given time was up

That's a very common thing which happens during live demo from what I heard from more experienced presenters. I feel I need to learn a lesson from this:

• Always check that projector works fine with laptop
• It might worth to buy external video adapter, just in case.
• I should swap timeslots with next presenter allowing myself more time to prepare environment
• Demo environments should have backups

That's it, folks! (c)
I'd like to have the same talk another time :)

I hear the same statements and misunderstandings over and over again from people who like dynamically typed languages. Obliviously, that spikes a lot of endless conversations and fights between two camps.

Generally, I'm avoiding conversations about static vs dynamic typing but every once in the while I drifted into that and hear the same statements, all the time. Often both sides just aren't able to listen to each other and thus conversation end up in the dead end.

Let me lay out those statements

It's faster to start to code!

I'm sure it is. It's cool if it's possible to fit the "hello world" application into a tweet but how useful is that? If one really cares about it - helloworldcollection.de.

Normally we create applications for businesses and they're always slightly complex than "hello world". We as developers not only create apps, we work in the teams, we support our code - it has to be maintainable.

Dynamic languages are cool for MVP

I have to agree, at least partially. Whenever business wants to validate hypothesis we build a simple implementation using whatever we find around and glue altogether somehow. The biggest mistake I see that people take it as a base and continue to improve that. In most of the cases it has to be throw-away code. Unfortunately, it doesn't work this way. Weak foundation will fire back very soon and instead of developing product tech teams fights with legacy and building workarounds around crutches.

In other words it's very difficult to scale your project's codebase without aid from the compiler and IDE inherently.

Types make you busy thinking!

I don't have time to think!

It's easy to start working without thinking about how data is organised, business domain, and how it flows through components(especially if there is no implementation yet). It's uneasy to think about that ahead but we have to do so at some point. If all rules are known from the beginning it's way easier to organise the structure.

Types really help in this place. They allow to draw a bold line and add meaning to the data, specify contract.

Oh wait, did you just complained that your employer makes you think of what you're doing?

Contracts

What if you ask glass to bark?

What happens when you drop an apple? Or an Apple MacBook? Since the same rules apply to every object we know what to expect from our actions. The last thing you expect when you dropped banana is to become coffee and fly to the stratosphere.

On another hand, you can't drop the green colour. It simply doesn't make any sense. Why would you allow do to so? Without contracts with known types and strict checks developer can likely can make a mistake without intention and world can collapse. Types can help to prevent that without exposing the dangerous situation to the end customer who, you know, pays money. One can say that those corner-case situations can be covered by unit tests.

Unit testing

Necessary evil we deal with

I sincerely believe that unit-testing is evil. We should avoid it as much as we can. For most of us who aren't using something like Haskell, the reality is very different. Due nature of the complexity of the systems we design, lack of math education(probably PhD) we simply can't comprehend all variations and ways program works. On another side modern programming languages have very limited support due to the complexity of implementation or lack of demand, I suppose.

That all make us write unit tests. It's a good compromise between creating very complex programs and being confident that things work as expected.

One might have noticed that the world isn't black and while. While we still have to write unit tests it would be nice to have some aid from the compiler.

Say, we have a contract which takes a specific type of object and returns something nice. Why would we write a unit test which supplies array instead if it's simply can't be a case, like at all? Contract with types can significantly reduce the number of tests have to be written and brain power to be used since the compiler will check that for us

We are humans

sin((5452 * 3023) / 2342.23053)⁴²

It's an interesting observation, isn't it? We do creative things, talk to the businesses, understand requirements and satisfy business, get paid for that.
Problem with humans that we don't really have much capacity to fit complex systems on different levels. We can easily make stupid mistakes while focusing on the high level, and that's okay.

Developers don't write code. Developers help businesses to solve problems. Focus on the result and deliverables are the most priority, implementation - is just a detail. I hope this fact doesn't hurt you much :)

In my understanding, we should focus on the business problem solving not on the low-level fights with code. Why do I even need to think what happens if string passed to the function which takes a number? I'd prefer a computer to check those things for me. That's a nature of automation.

So what?

¯\_(ツ)_/¯

I want to believe that everyone chooses the right tool for the task. But sometimes we simply can't see options as we didn't know about them. I appreciate that many people came to the industry recently not only without a CS degree but without high education at all. I think it's beautiful that nowadays it's possible! Also, my appreciation for ones who had the courage to change the industry - you're the best.

I hope this article can help to reconsider the use of tools which supports types - for better you!

Interesting fact.

For more than a decade, I was observing how the industry of dynamically typed languages changes their approach to handling big projects. Here is a list of popular projects gaining huge adoption for  type checks  in dynamically typed languages:

• PHP - built-in from version 7.0
• JavaScript - TypeScript,  flowjs
• python - PEP 484
• ruby - Sorbet
• groovy - @CompileStatic

I'm not saying that every user of those languages is adopting type checkers but from my observation, it's getting huge traction especially last 3 years.

disclaimer: if you think that technology works for you and does what you expect it to do - you're doing the right thing.

The talk took place in Spaces, Reading on 21^st of February 2019 and was organised by Google Developer Group Reading, namely by Chris Guest and his team.
In this talk, I highlighted reasons for learning one more language, guided through most noticeable features. In the second part, we had a live demo - kwitter REST web server.

I found people there were very friendly, it feels like they know each other for a long time. Really nice atmosphere and good vibes!

For me, it was the first time when I was talking for two hours. I think it went not terribly bad, at least I saw some people engaged in the demo, some questions were asked. If you were there - I'd love to hear the feedback!

My presentation with code snippets:

Demo project is available too - https://github.com/ruXlab/kwitter-demo

Thank you for attending the event, I hope you've learnt something useful.
And big kudos to Chris who helped me to prepare a presentation structure and gave very practical bits of advice.

Cover picture for this post is taken by @akostarevas

If we do 3+4 we always expect to get 7. Call to createDatabase shall not destroy database. As experience grows developer grasps more and more concepts and approaches because of the past experience and logic. It's very important part of programming ecosystem which helps to grow skill set without getting another Masters degree or attending classes/courses

People ended up with very common concepts and gave them names - algorithms, design patterns, data types, naming conventions.

I've used dozens of different languages for different platforms such as (android, web, desktops, symbian, j2me, embedded AVR & ARM, etc..) and I see quite the same patterns and approaches pretty much everywhere, almost. But JavaScript is always stands out!

Add mayo to the socks

I won't mention about hieracy system in JS - it's a mess, nonsense and you probably know about that. I want to point out on another amusing parts of the language.

> {}+[]
0
> []+{}
'[object Object]'

:thinking_face: Generally in the real life and normal languages you won't sum up chairs and color because they are too different. I'd except to get error, ideally compile time error because this code is nonsense. But no, JS actually executes it and returns result (different result, you woulnd't expect that, whould you?)

It's not true, because of "soda"

Boolean math is the foundation of the computers. Literally.
As a type Boolean is one of the most important way to express conditions and flags, binary state(means it has two states)

disclaimer: it's not only JS problem, most of weak-typed languages have that nonsense

> true == 0
false
> true == 1
true
> true == 10
false
> true <= 0
false
> true >= 0
true

So wait, true is not 0 but 1. My common sense says me that false should only correspond to 0 and true - for any other value, because you know, it's boolean. But JS engine is cheating here as well. For two state value we have an integer range lying in the range of true 𝈨 (-Min, 0) 𝉅 (0, +Max). Is it only me who founds it weird?

Obviously, my initial understanding of how to compare apples to sheeps might be wrong. But how can we use programming language which allows us to refer to the gut feelings?

Nono no, it's 'true'

There is an binary operator which makes sense only in context of boolean or group of booleans - not. What it does it simply inverts boolean value or bits in the number

But wait, in JS there are more usages!

> !""
true
> !"rabbit"
false

According to JS architects it's okay to perform negation on the string. If they could really try to use their imagination they could come up with real negation/inversion operation which makes sense to given datatype, for example by performing !"manDARIN" -> !"MANdarin".

Boolean not in JS inverts string and produces boolean. If string is empty - it's true, otherwise it's false. That's strange but okay, may be it's supposed to be a shortcut to check if there is something? If so why not to call string.isEmpty(), for example?

> !nonExistingString
    ReferenceError: nonExistingString is not defined
        at repl:1:2
        at ContextifyScript.Script.runInThisContext (vm.js:23:33)
        at REPLServer.defaultEval (repl.js:339:29)

Oops no, I mean, !"oops"

Who said that JS is easy language?

Strings are easy dangerous

Typical JS application operates with massive number of strings, so I'd assume toolkit for this data type should be perfect

Strings are almost arrays

Just recent discover. You would think you can access chars of the string the same way as you do in the vast majority of the languages using indexed access, but..

> var s = 'abcd'
undefined
> s[0]
'a'
> s[0] = 'X'
'X'
> s
'abcd'

Basically, it is possible to extract character at given position AND assign something to the specific position BUT it DOESN'T WORK. How do you feel about code which doesn't throw any error AND doesn't work, by design. Difficult to believe that it's in language spec

String literals aren't really strings, sort of

Let me impress some professional developers - there are string literals and string object, which behave, almost the same way. Something tells me some developers even didn't know there is a difference between.

Consider example from SO:

var x = "hello" ;
x.y = "world";
console.log(x.y); // this will print undefined

var x = new String("hello");
x.y = "world";
console.log(x.y); // this will print world

One can say, it's completely fine as they're different datatypes, even it's dynamically typed language. Generally speaking I hate inconsistent behaviour but if it significantly improve performance sometimes we have to go with this kind of tradeoff. My main issue here with error swallowing. Lovely javascript doesn't indicate that property wasn't set, even in strict mode.

Map/Set are useless

If you like to use existing data structures you also might use Set and Map, which finally were introduced in JS6. I really was waiting for that and had to use lodash before

But don't celebrate it yet.

Let me tell you story about Iterators. There is a design pattern which allows consumer to walk through collection(basically list of items organised in some way) and get every next element, only one element an iteration. In functional languages and pretty much everywhere nowadays it's common to use Iterator interface to perform functional-like operations on collections, such as map, filter, reduce.

I'm happy to confirm that these operators were around for a while for Arrays. But, think about it, in 2017 after ten years of working on new version architects didn't bother to add these operations on all-new Map and Set. Iterators there are half-backed and don't behave as you might expect.

You simple can't transform or filter key-value pairs on it

var m = new Map() 
m.set("a", 11)
m.set("b", 22)

> m.filter
undefined
> m.map
undefined

You might think I've forgotten about entries(). Well, look at this nonsense:

> m.entries()
MapIterator { [ 'a', 11 ], [ 'b', 22 ] }
> m.entries().map
undefined
> m.entries().filter
undefined
> m.entries().reduce
undefined

So what is a javascript way?

If you understand roughly what is happening here you might be terrified:

> new Map(Array.from( m ).filter(([key, value]) => value > 20 ))
Map { 'b' => 22 }

Same thing is happening with Set:

var s = new Set([1,2,2,3])
> s.map
undefined
> s.filter
undefined

And the recommended way to perform operation on collection is using Array:

> new Set(Array.from(s).filter(v => v >= 3))
Set { 3 }

When I see those unnecessary allocations due transformations to array and back to map I'm shivering. It immediately became known anti-pattern and no one uses this approach. Apparently because of those issues developers consider neither Set nor Map to use. :sad_face:

The good news both Set and Map do have forEach function which returns iterator with pairs of key-value wrapped with array, if you still care after all:

> m.forEach((k, v) => console.log(<code>${k}=${v}))
11=a
22=b

As I love to say those data types are implemented in JavaScript style.

But wait, how do people write so many apps and they seem to be working?

The answer is simple:

1) Luck. Yes, sometimes we're lucky enough to make app which doesn't fail for years because corner case never comes. It doesn't mean it won't happen. The more system in use the more likely hidden bug will be discovered in the next moment, as per Bayes' theorem. Also, having one failure of thousand requests can be considered "okay".
2) Unit tests. The biggest unit tests and TDD adopters are found in javascript community. I believe unit tests are important, unfortunately. Many of them simply cover what compiler can't infer itself - practically we complement compiler's stupidity/simplicity paying high cost of time of our human lives. On another extreme in Haskell world you won't find much unit tests since compiler guarantees really a lot.
3) Good practices. They exists in any ecosystem and they guide developer to write good, readable code, solve well-defined problems in common way. In javascript world good practice is more of "must use practice" otherwise system explodes. Even you can you should never assign property to string object, because it could be literal. Instead pollute namespace via prototype use helper functions, because, you know, js is not fully classic OOP language ¯_(ツ)_/¯

Instead of conclusion

I always say if technology works for someone I don't mind what is being used until business is happy. But looking on javascript ecosystem I feel it drains a lot of human energy, which could be used in more conscious way in another worlds. Developers often fight windmills instead of bringing more value to business.

As a business owner it's not always easy to handle huge volume of calls 24/7. On another hand each customer is important and it deserve to be served well.

To kick off development you can checkout demo repository

What to expect in this tutorial

In this tutorial we build simple voice mail forwarder where callers asked to leave a voice message which will be sent to the email using Nexmo Voice API as an attachment.
Example of result:

That can be done just in a few steps:

• Configure environment
• Create a voice application and assign it to the phone number using Nexmo CLI
• Build a simple Spring Boot application
• Handle call events
• Handle recording webhook event and email record

Prerequisites

• A Nexmo account
• Demo Source code
• The Nexmo CLI installed and set up

Configure environment

Service has to be available from the internet. The easiest way to do so is to use tunnel such as ngrok or servio.net:

ssh -R 80:localhost:8080 serveo.net
Forwarding HTTP traffic from https://iste.serveo.net

It will forward all incoming traffic from the randomly assigned host to your machine to port 8080.

Create an voice application

Next step is to create application associated with this service

➜  nexmo app:create "voicemail app" https://iste.serveo.net/answer https://iste.serveo.net/event \
             --keyfile nexmo_app.key
Application created: 0d76428d-a914-4d65-8c38-814d76986391
Private Key saved to: nexmo_app.key

Afterwards we can purchase phone number and link it to the app

➜  nexmo number:buy --country_code GB --confirm
Number purchased: 441414658675

➜  nexmo link:app 441414658675 0d76428d-a914-4d65-8c38-814d76986391
Number updated

Spring Boot Application skeleton

In this gradle file apart of spingboot itself we also need two more dependencies: spring-mail for SMTP client and nexmo-java library which simplifies access to the Nexmo APIs

dependencies {
    . . . 
    compile 'com.nexmo:client:3.3.0'
    compile "org.springframework.boot:spring-boot-starter-web:2.0.0.RELEASE"
    compile 'org.springframework.boot:spring-boot-starter-mail:2.0.0.RELEASE'
}

Main application class is very basic but we need to add a bean factory for NexmoClient for further handy access:

@SpringBootApplication
public class VoiceMailSrv {
    public static void main(String[] args) {
        SpringApplication.run(VoiceMailSrv.class, args);
    }

    @Bean
    public NexmoClient nexmoClient(
            @Value("${nexmoApplicationId}") String applicationId,
            @Value("classpath:nexmo_app.key") Resource cert
    ) throws IOException, NoSuchAlgorithmException, InvalidKeyException, InvalidKeySpecException {
        return new NexmoClient(new JWTAuthMethod(applicationId, Files.readAllBytes(cert.getFile().toPath())));
    }
}

Here you might notice that we need to provide applicationId and key, generated using nexmo-cli. To do so put nexmo_app.key to the resources folder and create application.properties file where spring boot will pick up application configuration.

In resouces file you need to provide applicationId and additional configuration for SMTP server:

nexmoApplicationId = 0d76428d-a914-4d65-8c38-814d76986391
destinationEmail = bob@email.com
senderEmail = me@localhost


# Email server configuration
spring.mail.host=smtp.gmail.com
spring.mail.port=587
spring.mail.username=<login user to smtp server>
spring.mail.password=<login password to smtp server>
spring.mail.properties.mail.smtp.auth=true
spring.mail.properties.mail.smtp.starttls.enable=true

As it normal spring configuration it can be provided/overloaded via environment variables, command line arguments and many other options

More information about SMTP configuration can be found in blog post on Baeldug

Handle call events

When Nexmo receives an inbound call to purchased number it makes a request to the webhook endpoint you set when you created a Voice application. There are two of them - AnswerURL used to provide NCCO Object and EventURL for events happening throughout call lifecycle

To instruct Nexmo about actions we need to return NCCO Object when we recieve call on AnswerURL, our controller handler should look like:

In our case we design NCCO of following actions:

• Talk - Using Nexmo TTS say "Thanks for calling us. Please leave your message and press hash key"
• Record - Record a message and put it on Nexmo Media service. In this action we need to provide callback url to be called when voice message is recorded
• Talk - Speak "Thank you. Your message has been recorded. Bye"

@GetMapping("/answer")
public Object onAnswer(
        @RequestParam("conversation_uuid") String conversation,
        @RequestParam("from") String caller,
        HttpServletRequest request
) {
    log.info("onAnswer: got call from {} (conversation {})", caller, conversation);
    return new NccoResponseBuilder()
        .appendNcco(new TalkNcco("Thanks for calling us. Please leave your message and press hash key"))
        .appendNcco(new RecordNcco() {{
            String schema = Optional.ofNullable(request.getHeader("x-forwarded-proto")).orElse(request.getScheme());
            String host = Optional.ofNullable(request.getHeader("host")).orElse(request.getLocalAddr());
            setBeepStart(true);
            setEndOnKey('#');
            setEventUrl(String.format("%s://%s/record", schema, host));
            setEventMethod("GET");
        }})
        .appendNcco(new TalkNcco("Thank you. Your message has been recorded. Bye"))
        .getValue().toJson();
}

No need to craft NCCO manually - it can be done by using fluent API.

Also, we need a SMTP server. There are plenty of options such as sendgrid or mailgun.

Handle recording webhook event and email record

When recording is finished the specified callback for record action will be called with url to mp3 file. Our handler should download it and compose an email attaching it.

<br />@GetMapping("/record")
public void onRecordingFinished(
        @RequestParam("recording_url") String recordingUrl
) throws MessagingException, IOException, NexmoClientException {
    log.info("onRecordingFinished: new recording detected: {}", recordingUrl);

    Recording recording = nexmoClient.getVoiceClient().downloadRecording(recordingUrl);
    byte[] binaryRecording = IOUtils.toByteArray(recording.getContent());
    log.info("onRecordingFinished: recording has been downloaded, {}bytes", binaryRecording.length);

    MimeMessage message = emailSender.createMimeMessage();

    MimeMessageHelper helper = new MimeMessageHelper(message, true);

    helper.setFrom(senderEmail, "Voicemail demo service");
    helper.setSubject("Your voicemail recorded at " + new Date());
    helper.addTo(destinationEmail);
    helper.setText("Please find voice recording in attachment");

    helper.addAttachment("Recording.mp3", () -> new ByteArrayInputStream(binaryRecording), "audio/mpeg3");

    emailSender.send(message);
    log.info("onRecordingFinished: email has been sent to {}", destinationEmail);
}

Media storage is protected by JWT authorization but NexmoClient simplifies access to it and returns InputStream we can work with

The full code of controller can be found on github

Conclusion

In this tutorial we quickly built simple voicemail recorder using seamlessly integrated nexmo-client to the spring boot application

Nexmo has various actions in NCCO and combining them together it is possible to create endless applications for different use cases.

It was first experience of this kind. Although I already had number of public talks related to kotlin this one was in new format for me - live coding. After short introduction about myself and kotlin language I started to show off main features of the language while developing REST notebook service using Spring Boot. I found it's unfair that kotlin is getting associated more with Android while it's actually very good on backend too. More importantly - not only as better replacement for enterprise applications but also for prototypes - it easily can compete with python and nodejs for PoC/MVP projects

Here are slides

The source code is available at github
https://github.com/ruXlab/kotlin-springboot-demo as well as script for scenario

DISCLAMER
1. This article is about JavaScript - the language happen to become a runtime for many types of applications.
2. TypeScript, CoffeeScript, ^*Script have nothing to do with JS, this post is not about it.
3. As many of you I'm also a JavaScript developer. I'm not addressing to anyone personally, the post below is just my opinion about technology
4. If you're happy with any technology and it helps to archive the business goals - I can be only happy for that!

Big claim, isn't it? Well, I'm sure I have a right to say that. I've been using suffering from JavaScript in last 10+ years. Really, I can't remember anything good about this shit language since I started to make money on programming being freelancer. I'm not mean, I just had enough. And please, don't take this article personally, you're not a programming language you use.

I'm a normal developer.

It means I'm not genius and I can't and wouldn't keep whole program code and it's complicated flow and branching in my head. I'd rather delegate this boring work to computer and focus on business logic as much as I can. I'm certain, there are about 2.5 really good developers who can write good code in javascript. But probably they don't as they are busy talking on conferences ¯_(ツ)_/¯

You cannot avoid JS

Yes, you can try but you won't be able to. As a cancer it's being integrated in everything - web, backend, databases, mobile and embedded systems.

On frontend we don't really have any option - js ended up as the only tool for programming in your browser(I don't count dart, it's dead as virtual machine). It's far from perfection and that's why ecosystem created hundreds of different languages on top of it - coffeescript, typescript, etc.. They are actually transpiling to javascript. Even new versions of JS - ECMA6 still needs to be transpiled

Again, due historical reasons browsers don't have other options, suck it up, bite your tongue and write frontend. I know it's a hard work, thanks to people who doing that.

JS is cool on backend too

:facepalm:
I hear that all the time, especially from newbie developers. They're claiming that JS is so simple to use so even backend can benifit.
There are dozens of tools have been created for backend - JVM/Java, RubyOnRails, Python and, I'm sorry, PHP.

So why do they think we deserve one more creature in our crowded backend zoo?

I'd like to go through most popular answers I've heard

JS is fast!

Sorry, comparing to what? How do you measure? I'm not surprised anymore when mature JS developers have never heard of costs of context switching, for example. Or maybe one heard about new mutlithreading in javascript? Oh c'mon you serious?

JS is modern language

Oh well. I want to share pain with you.

I'm sure you have heard about JS6, along with cool systax stuff they finally brought in classic datatypes - Map & Set. I hope dear reader know these are very basic data structures known to any developers(even to JS ones, I hope). I was really happy that finally I could use innovations dated back to 1970

But here is a catch. Being so-so developer I expect collections to be iteratable same as Arrays in javascript, not a bit deal, isn't?

You easily can do [1, 2, 3, 4].map(i => i*10) because it's so normal nowadays, and, of course, JS clamed to be functional. But, apparently JS "architectors" don't think so. Althogh Set has forEach method but doesn't have map, filter, reduce, etc...

I mean, WHY?! What held developers from doing that?

Same with Map - it's simply not possible to map via key or entries even forEach works fine. But JS community doesn't care, it's okay to write another leftpad or money-patch global objects(or classes? Or functions?). It's even interesting! Writing more code it's always fun, isn't it? You can find more amusing things in "javascript vs logic" post.

Type system

I was told that types slow down development. And I'm absolutely agree, if you write Hello World every day JS is hell faster to use. But sorry, I'm writing production code for medium and big sized applications, which are business critical.

They funny thing is while application is growing it increases amount of errors and tests for JS, while for staticly-typed languages(hello JVM) code maintance complexity stays about the same. It takes time and experience for JS guys to understand that, and when they finally open eyes shit started to happen. They are pulling TypeScript and flowjs to make code SAFER, basically trying to make remove script part from JavaScript. It always amuses me

Tests

One claims that JS is safe if you write tests. It's true, but why whould you write tests if compiller can prevent developer from obvilius mistakes? I mean, why? Why do you need to check input arguments for type(say, MongoId as object or string) if you expect only one type?

I'm sure it's easier to write id: MongoId in the contract rather that cover that scenario in the dozens of tests(and still it will leak some usecases)

I love reckless approach of some JS developers - we have tests covered that - they say. Often I don't have tests I don't need - compiller won't allow me to run something which doesn't work

Again, I see zero reasons why you would run broken code, you should not be able to!!

JS is envolving all the time

Actully it's other way around. Because of the low learning curve we're having a lot of shit code outthere(same as PHP). Honestly it looks like there is a rally of producing another shit-framework.

While JS devs are bragging about production deployment of all-shinny-new library released yesterday world is shocking observing that. Why so? It might sound a bit conservative but production code supposed to work.

It's very easy to write code in JS

Yes, it's always easy to write code, but don't you write Programms and Applications? It's not that simple, and, if tool doesn't put constrains on you it's very difficult to keep consistency.

Think about writing code on the sheet of paper? It's very simple but never easy

So what?

If you're frontend developer - accept it, suffer and do your job. It's hard but very important.
If you're JS backend developer - stop bragging about that. It's not fine. Open your eyes as look around, world of programming is beatuful and got different tools, don't reinvent the wheel. I know you can do better.

Dude, do you use JS?

Of couse I do.

I'm just minimizing usage to keep mine nerve system for more pleasant experience.
Yes, I do write frontend (and do it well) when I have to, and yes, I do write prototypes on nodejs(but it never ever goes outside of prototype stage)

Why so angry?

Coz I'm sick of observing how JS is holding back whole software development ecosystem. As part of it I have my word.

The goal of this article is to doubt your (most likely) blind love to JavaScript and look around in the Bigger world

I had a talk "Data Mining Toolbox" in University of Cantabria, Santander, Spain organized by meetup group "Data Science Santander" on November 9th, 2017

It was my first public presentation on this topic where I accumulated some my experience in html data extraction and processing.

Code snippets used in presentation

For convenient access I put all code snippets below instead of creating new repo on github

Basic example with reddit

$('.thing.link').map((k, i) => {
   var i = $(i);
   return {
       title: i.find(".title a.title").text(),
       url: i.find(".title a.title").attr("href"),
       score: i.find(".score.unvoted").text(),
       subreddit: i.find(".subreddit").text()
   }
}).toArray()

And

window.open().document.write(JSON.stringify(a))

To create new browser window and put there json representation of the javascript array

Pagination in reddit

As it was shown it is possible to parse not only currently loaded page in the browser but get number of the pages

function _page(n, afterId) {
   $.get("https://www.reddit.com/?count=10&after=" + afterId, data => {
       var result = $(data).find('.thing.link').map((k, i) => {
           var i = $(i);
           return {
               id: i.attr('data-fullname'),
               title: i.find("p.title a.title").text(),
               url: i.find("p.title a.title").attr("href"),
               score: i.find(".score.unvoted").text(),
               subreddit: i.find(".subreddit").text()
           }
       }).toArray()
       console.log("page", n, "=>", result)
       a.push(result);
       if (n <= 0) return;
       _page(n - 1, result[result.length-1].id)
   })
}

Code snippet from real project

Code snippet bellow potentially can get extract from the StackOverflow job page including tags and optional salary. Published for educational purposes only

    fun fetchRecentPage(): List<StackOverflowJob> {
        val jobs = jsoupGetDocument(url).select(".jobs .listResults > div[data-jobid]")
        return jobs.map {
            val job = StackOverflowJob(
                wipeoutHtml(it.attr("data-jobid").trim()),
                normalizeUrl(wipeoutHtml(it.select(".-title a.job-link").attr("href").trim())),
                it.select(".-title h2").first()?.text()!!,
                it.select("p.description").text()
            )
            job.location = it.select("li.location")?.text()?.trim()
            job.tags = it.select(".tags a").orEmpty().mapNotNull {
                it.text()?.trim()?.let { wipeoutHtml(it) }
            }
            job.company = it.select("li.employer").first()?.text()?.trim()
            job.salary = it.select(".salary").first()?.text()?.trim()
            job
        }
    }

In this example kotlin and jsoup are used

Topic of that talk was same as in London but reflected by received feedback plus some extra things.

Here is slides

All examples can be found at my repo github.com/ruXlab/safe-kotlin-features-showcase

The extra example was with typesafe units math. With kotlin it's possible to work with different dimensions in very safe manner, so you won't subtract liters from wats. In my basic example I work with distance, time and speed:

fun main(args: Array<String>) {
    println(10.asSeconds + 5.asMinutes)
    // 310.0s

    val speed = 10.asKilometers / 5.asHours
    val speedInMPerSecond = speed.to(Meters / Seconds)
    println("$speed = $speedInMPerSecond")
    // 2.0km/h = 0.556m/s

    val speedInMPerHour = speed.to(Miles / Hours)
    println("$speed = $speedInMPerHour")
    // 2.0km/h = 1.2427454732996135mi/h

    val `50mph` = 50.asMiles / 1.asHours
    println("$`50mph` = ${`50mph`.to(Kilometers / Hours)}")
    // 50.0mi/h = 80.467km/h
}

From km/h to mi/h - easily! Implementation might seem difficult to understand for someone isn't familiar with kotlin syntax alot, so I broke up it to step-by-step instructions

According to meetup.com there were ~100 people while I'd say ~80. After talk I was asked a lot of questions starting from presentation's topics ending up with testing, frameworks and some hate of php and js of course.

And yeah, I already know what is the next topic I want to present :)

You probably know that I really care about productivity, hence writing safe code is very important part of software development for me. I want compiler check things during compile time not to have exception in production when business can suffer. I want to code business logic not reinvent checks and write tests(especially unit tests)

There is no video recorded but I have my presentation shared:

Also, all code snippets are available on github ruXlab/safe-kotlin-features-showcase

Personal note

It's been a while since I had public presentation.
Before I had only one public presentation in London - at Nexmo, which was internal, only for my colleagues. It was about kotlin as well
That's it. Back in Saint-Petersburg I had a lot of public talks, sometimes even few times a week. Hopefully I can return back on track :)

I really like the way office scanners in multi-functional devices work. Normally if you want to scan you just load stack of paper into and put your email address. Scanner does the rest and in minute you'll get ready-to-use pdf file in your inbox.

I was thinking about having button attached to RPI Zero which initiates scanning and document upload.

Alternatively it's possible to program another/one more action, for instance document copy. It's really up to developer - Raspberry PI provides you endless possibilities!

Let's build this scan button:

• It should indicate scanning process right after user hit the button. The problem is the scanner has to be pre-heated to operate and it can take up to 20-30 seconds. User shouldn't be puzzled by current process.
• To display current process we need LED. I found yellow LED 3v
• Scanned document should be placed in shared folder accessible from all popular OS. There are not so many options, hence we have to use Samba(windows files sharing service)
• Documents older than 1h should be removed from that shared folder
• Scanned document format - jpg

Prepare environment

First things first - we need python libraries and samba server

sudo apt-get install samba
sudo apt-get install python3-gpiozero python-gpiozero python-pkg-resources

Button script

The script is fairly simple.

It configures LED and Button ports. Then listens to button event. When button is clicked LED blinks to indicate process start and script excecutes scanimage utility which convert output from ppm to jpg format.

<br />from gpiozero import Button, PWMLED
from signal import pause
from time import sleep, strftime
import os;


OUT_FOLDER="/smb/scanner"

led = PWMLED(15)
button = Button(23, pull_up=True, hold_repeat=False)

def blink():
  for i in range(20):
    if i % 2 == 0: led.off()
    else: led.value = 0.3
    sleep(0.1)
  led.off()

def toggle():
  blink()
  led.value = 0.3
  ts = strftime("%d-%b-%Y %H.%M.%S.jpg")
  cmd = "scanimage --resolution 150 | convert - jpg:- > '{}/{}'".format(OUT_FOLDER, ts)
  print("Scanning... {}".format(cmd))
  os.system(cmd)
  led.off()


button.when_released = toggle

pause()

PWMLED is used in this script on purpose. Seems like I bought too powerful led without any specs - lady in the shop ensured me that it's low current LED. Unfortunately one of GPIO port has died after I left it on for 10 minutes. To avoid burning another GPIO pin I enabled PWM for LED, if you never heard about it read wikipedia. From my experiments I found brightness of 30%(0.3) seems minimum acceptable for me. You might need to connect your led via resistor, depending on it's current and voltage.

I couldn't run this script under non-privileged user unfortunately because scanimage command doesn't detect scanner. I think there were problems with USB device permission but if you care I'm sure it's possible to fix it.

Done

I forgot to copy other configuration files for samba server, crond and systemd and now I don't have access to RPI but trust me it used to work :D I'll put here other configs later on

This blogpost has been written in airplane over Baltic sea on the way to London

What if I say it's possible to predict 700 values of this line using just:

• 30 data points for feeding neural network(which is just half-period)
• using just two fully-connected layers (hence it's not deep network)
• having just three neurons in whole network

I was quite surprised how fast NN could learn from such a small input even without "seeing" one full period. In previous experiment I trained network providing actual argument of that function as input of NN.

So what is the difference? In this experiment you can find method dataset() which generates pair of (X, Y) from the actual function values. It's something similar to "sliding window" over function's values. For instance if our function f() produces values (n₁, n₂, n₃, n₄) then dataset to be fed to NN looks like (n₁, n₂), (n₂, n₃), (n₃, n₄).

Show me output!

Red line is what our NN has predicted. Barely visible light blue spikes - true value. Not bad for just 3 neurons, huh?

In the first cell of notebook you can play with hyperparameters (SAMPLES, EPOCHS, BATCH_SIZE) and see differences in the result. Default ones aren't optimal thats why you can spot light blue line on the chart. Increasing amount of input datapoints and/or number of epochs will dramatically increase precision quality and MSE → 0

Open Experiment In GitHub

For other experiments using keras check out this 1st post

I've got freelance job to write Alexa Skill so I decided to play around with pet project. It's London Bus Stop - custom Alexa Skill which helps user to get information about busses coming to saved bus stop.

How to install it?

Go to London Bus Stops and get this skill or just say Alexa enable London Bus Stops skill.

How to configure it?

TL;DR;

Go to TFL website and search for bus stop you want to use. On the stop's page you'll find a short 5-digit code(same one you can find on the sign plate):

After that say «Alexa, ask London Bus Stops for setup». You'll be asked for that 5 digit code you have to verb and confirm. Sometimes Alexa Platform can't understand what number you meant so try different combinations. For 48584 if could be 48-58-4, 4-85-84 or 48-584 and so on.

Why configuration is so difficult?

This is very tricky question. Due Alexa platform restrictions it's nearly impossible say something like «Remember my bus stop is Old Street» because:

• Old Street has 4 pairs of bus stops
• Each pair has 2 directions
• Platform restrictions don't allow to use speech-to-text
• If skill found few bus stops it's annoying to use voice interface to make a selection

After some experiment I ended up with most reliable way to setup bus stop using short code. If you have other ideas I'm very open to it!

How does it work?

Thanks to UK Open Data Portal and very well done Transport For London API.

Server is written in python using flask-ask. It's very nice unofficial library of decorators for flask. Code looks very concise and decorators make it looks like it's java annotations :)


@ask.intent("ConfigureIntent")
def configure(is_repeat = False):
msg = render_template("configure" if not is_repeat else "configure_repeat" )
session.attributes['mode'] = "configuration"
return question(msg)


I have found and prepared database with shortcode, longcode and busstop name which is used for search and real time updates queries. Every time user ask for update skill find associated longcode stop id and make query to TFL API, process response and reads the result

Why did I create it?

Even I don't have Alexa at the moment I hope Londoners will find it useful in daily life :) I might publish it to open source as it free anyways