Dota 2 Stats Crawler #1: Project Setup

First step will be creating a hello world style entry point which will be put into a fat jar later.

So code for that is trivial and it’ll be placed in src/main/scala directory following maven convention of Java projects.

CLIMain.scala

package pl.stmi.dota2.crawler.main

object CLIMain extends App {
  println("Dota 2 Stats Crawler")
}

Now that there’s entry point for application let’s configure building and packaging.

For build automation sbt tool will be used and sbt expects build configuration in build.sbt file placed at project root.

First lines define top level project parameters like name, organization and version, also Scala version is explicitly defined. After this configuration of sbt-assembly plugin is provided. Initially it consists only of final fat jar name and qualified name of mainClass - CLIMain in this case.

build.sbt

ThisBuild / name := "dota2-stats-crawler"
ThisBuild / version := "1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.2"
ThisBuild / organization := "pl.stmi"

lazy val dota2_stats_crawler = (project in file("."))
  .settings(
    assemblyJarName in assembly := "dota2-stats-crawler.jar",
    mainClass in assembly := Some("pl.stmi.dota2.crawler.main.CLIMain")
  )

In order for this to work sbt-assembly has to acctually be added as a plugin to a build. In sbt it’s done via placing files with *.sbt extension under project directory. By convention it can be named plugins.sbt, but actuall name is irrelevant any file with *.sbt extension will be picked up by sbt.

plugins.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")

Also in project directory sbt version can be fixed for build reproducibilty. This is done by creating build.properties and setting right props.

build.properties

sbt.version=1.3.10

After this invocation of sbt assembly command from terminal should produce a runnable jar in target folder. Unlike in Java projects it won’t be placed directly in target, but rather in appropriate scala subdirectory scala-2.13 in this case.

It should look something like this

> sbt assembly
> java -jar target/scala-2.13/dota2-stats-crawler.jar
Dota 2 Stats Crawler

Let’s also make sure that no content generated in build will be accidentally added by defining .gitignore rules for repo. Those will be fairly short - just pass over anything in target and don’t include IntelliJ project files and dirs.

# SBT/Maven output folder
**/target

# IntelliJ IDEA config files and dirs
.idea/**
*.iml

Last and a little bit complicated step will be setting up CI pipelines configuration for GitLab. Gitlab pipelines configuration should be supplied in .gitlab-ci.yml file placed in repository root.

Starting point is defining default options for pipelines.

Begin with selecting image that will be used to run jobs in pipelines. Value should be a name of Docker image chosen for use in build stages. For this project image based I’ll use one based on Alpine Linux which is optimized for running in containers. Particular build will be from OpenJDK as it has OpenJDK 8 included, so that code and build can be run.

Next is before_script section where actions preceding each stage will be defined. In this case testing repo for apk needs to be added and sbt installed from there, as unfortunately sbt still isn’t included in stable Alpine release.

As last point set cache to keep dependencies downloaded by sbt between builds. This allows to save running time and bandwidth per pipeline thus allowing for better use of pipelines free minutes from Gitlab. Paths defined for caching are chosen based on sbt reference manual suggestions.

gitlab-ci.yml default section

default:
  # Use OpenJDK 8 Alpine image from DockerHub
  # https://hub.docker.com/r/_/openjdk/
  image: openjdk:8-jdk-alpine
  # Configure sbt installation before jobs, currently there's only sbt in testing
  # https://pkgs.alpinelinux.org/packages?name=sbt
  before_script:
    - echo 'http://dl-cdn.alpinelinux.org/alpine/edge/testing/' >> /etc/apk/repositories
    - apk add sbt
  # Indicate that those paths should be cached between builds
  cache:
    paths:
      - ".sbtboot"
      - ".coursier"
      - ".ivy2"

Next will be defining build steps, GitLab default stages shall be used. To preserve runnable jar build from assembly task artifacts parameter will be defined.

gitlab-ci.yml stages definitions

test:
  script:
    - sbt clean test

deploy:
  script:
    - sbt assembly
  artifacts:
    paths:
      - target/scala-2.13/dota2-stats-crawler.jar

Finishing touch will be making sure that there’s no ambiguity in definitions of directories to be cached, so env variables will be defined for pipelines.

gitlab-ci.yml env variables definitions

# Based on SBT Command Line Reference (https://www.scala-sbt.org/1.x/docs/Command-Line-Reference.html)
# define and cache directories for: sbt.boot.directory, sbt.coursier.home, sbt.ivy.home
# Use $CI_PROJECT_DIR predefined variable to avoid ambiguity in path definition
# (https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)
variables:
  SBT_OPTS: "-Dsbt.boot.directory=$CI_PROJECT_DIR/.sbtboot
             -Dsbt.coursier.home=$CI_PROJECT_DIR/.coursier
             -Dsbt.ivy.home=$CI_PROJECT_DIR/.ivy2"

That completes initial project work.

Links to project repo:

• Merge request
• Commit on master