Skip navigation

Category Archives: Linux


Just share a bit. When we want to commit source code, it’s better to check locally before pushing the source code to repository which then check them at the build server.

Here’s a source code for the wrapper script. The needed binaries are PHPCS, ESLint, and Stylelint.

To execute in the pre-commit, you can just use this.

SOURCE_FILES=${SOURCE_FILES:-`git diff-index --name-only --diff-filter=ACMR HEAD`}
export SOURCE_FILES
./checkstyle.sh

exit $?

I just watched The Last: Naruto The Movie a couple of days ago and I recalled that it’s been almost a quarter of decade since I last read the mangascan.

Reading Naruto in Kindle

Reading Naruto in Kindle

So I got a complete 2GB of mangascan pictures from my friend from volume 1 until volume 72. I’ve been a fan of Kindle e-book reader and I want to read the mangascan in my Kindle device. Here’s how I converted the PNG and JPG files to PDF so it can be readable in Kindle.

Basically what I need is imagemagick library to convert the images to PDF. Here is how to use the imagemagick to do the conversion.

As long as the JPG/PNG files is named with proper file naming, doing a batch conversion in zipped files is as simple as this. It shouldn’t take too long to convert all of the images to PDF. In my case it took less than an hour. The quality of the PDF is not that bad, but I think we can configure the imagemagick to increase the quality. I assume it reduces the quality by a bit.

for zipfile in $(find . -type f -iname "*.zip" | sort -n); do
    echo "$zipfile"
    filename=$(basename "$zipfile")
    filename="${filename%.*}"
    echo "$filename"
    unzip $zipfile -d $filename
    echo "Converting...."
    find $filename | sort -n | tr '\n' ' ' | sed "s/$/\ ${filename}.pdf/" | xargs convert -verbose
done

See? Pretty simple, right? (The fact that it’s really simple is why I posted here. Might be useful for me in the future).


For quite some time I’ve been looking for a very easy way to deploy a small service using Java that is accessible through HTTP and self-hosted, or in other words, it doesn’t need a installed server. I was initially thinking about using embedded Tomcat or Jetty by writing the server code manually.

But then it turns out that there is this project from Spring built specifically for this kind of application: the Spring Boot. As quoted from the web,

Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that can you can “just run”.

By using this framework, I only need to focus on building the service rather than writing the HTTP server. And running the service is just as simple as using plain old `java -jar`.

The whole source code can be seen in this URL https://github.com/petrabarus/springboot-opsworks-example.

I started by using the sample code from Spring Boot.

import org.springframework.boot.*;
import org.springframework.boot.autoconfigure.*;
import org.springframework.stereotype.*;
import org.springframework.web.bind.annotation.*;

@Controller
@EnableAutoConfiguration
public class SampleController {

    @RequestMapping("/")
    @ResponseBody
    String home() {
        return "Hello World!";
    }

    public static void main(String[] args) throws Exception {
        SpringApplication.run(SampleController.class, args);
    }
}

The skeleton code for the controller can not be much simpler than this.

And here’s the POM file.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.petrabarus.maleskoding</groupId>
    <artifactId>springboot-opsworks-example</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.1.6.RELEASE</version>
    </parent>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <!-- Sets to use Jetty. No particular reasons -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jetty</artifactId>
        </dependency>
        <dependency>
            <!-- For deployment ready -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
    </dependencies>
    <build>
        <resources>
            <resource>
                <!-- Creates directory for OpsWorks config -->
                <directory>src/deploy</directory>
                <targetPath>../deploy</targetPath>
            </resource>
            <resource>
                <!-- Creates config directory for external config --> 
                <directory>src/config</directory>
                <targetPath>../config</targetPath>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <!-- Creates JAR -->
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <!-- Creates zip distributable -->
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptor>src/assembly/dep.xml</descriptor>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>
</project>

The XML code below is the minimum configuration to run the service. (The spring-boot-starter-jetty is used to make the service runs using Jetty instead of the default embedded Tomcat).

<!-- cut -->
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.1.6.RELEASE</version>
    </parent>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <!-- Sets to use Jetty. No particular reasons -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jetty</artifactId>
        </dependency>

<!-- cut -->

To build the JAR, I use this plugin

<!-- cut -->
    <build>
        <!-- cut -->
        <plugins>
            <plugin>
                <!-- Creates JAR -->
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
<!-- cut -->

Building the JAR is simply just by executing

$mvn clean install

and the JAR will be in the `target` directory, e.g. target/springboot-opsworks-example-1.0-SNAPSHOT.jar I run it using the usual java -jar command

$target/springboot-opsworks-example-1.0-SNAPSHOT

I can see the result by using curl

$curl localhost:8080
Hello World!

Now it’s time to deploy it using OpsWorks. In this example I use `HTTP archive`, but it’s better to use `S3 archive` to provide better security since it can use IAM credentials.

Screenshot from 2014-09-20 00:02:21

The ZIP-ed file for deployment can be created using `spring-boot-maven-plugin`

<!-- cut -->
    <build>
        <!-- cut -->
        <plugins>
            <plugin>
                <!-- Creates zip distributable -->
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptor>src/assembly/dep.xml</descriptor>
                </configuration>
            </plugin>
<!-- cut -->

Here’s the assembly XML configuration

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2" 
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
    <id>bin</id>
    <formats>
        <format>zip</format>
    </formats>
    <fileSets>
        <fileSet>
            <directory>${project.build.directory}</directory>
            <outputDirectory>/</outputDirectory>
            <includes>
                <include>*.jar</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>${project.build.directory}/config</directory>
            <outputDirectory>config</outputDirectory>
            <includes>
                <include>*.xml</include>
                <include>*.properties</include>
                <include>*.yml</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>${project.build.directory}/deploy</directory>
            <outputDirectory>deploy</outputDirectory>
        </fileSet>
    </fileSets>
</assembly>

Now I can have a very simple build and distribution script like below. This can be use for Jenkins. When a new change is pushed to the git repository, Jenkins will download the change, build, test the source, and then pack and upload the binary.

$mvn clean install assembly:single && s3cmd put -P target/*.zip s3://path/to/the/upload.zip

As we know, deploying in OpsWorks is just a matter of click.

Screenshot from 2014-09-20 00:02:07

Note that deploying the OpsWorks app doesn’t mean running the service automatically. For that I need to use a deploy hook. Chef provides a way to execute a ruby script on steps in the deployment. Using the `before_restart.rb` I can execute a task that happens after the app directory changes its link from the old directory to the new deployed directory.

The content of `before_restart.rb` script is in below

script "runjar" do
    interpreter "bash"
    user "root"
    cwd release_path
    code <<-EOH
        java -jar *.jar > /var/log/springbootext1/app.log 2>&1 &
    EOH
end

I added the file in the `src/deploy` directory and I declare it as a resource in the POM file.

    <build>
        <resources>
            <resource>
                <!-- Creates directory for OpsWorks deploy hook -->
                <directory>src/deploy</directory>
                <targetPath>../deploy</targetPath>
            </resource>

The next question is how to shutdown the running service and update the new JAR. Spring Boot provides a production-ready feature called the actuator. Here I can put a shutdown for the service that will turn off the running service. To add the actuator, I define the dependency in the POM file.

<!-- cut -->
        <dependency>
            <!-- For deployment ready -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
<!-- cut -->

And in the deploy hook, before the run service script I added new script

script "shutdown" do
    interpreter "bash"
    user "root"
    cwd release_path
    code <<-EOH
        curl -XPOST "http://localhost:8080/shutdown" > /var/log/springbootext1/app.log 2>&1
    EOH
end

That script send a POST request to the current running service to shut itself down. As default the shutdown actuator is not enabled by default. I still need to add new configuration. To do this, I added a new file `scr/config/application.properties` and I added this line.

endpoints.shutdown.enabled=true

Spring Boot has a very flexible way of configuring the application. The simplest way is to have `config/application.properties` file in the JAR directory. The application JAR will check what’s inside the file and override the configuration.

The deploy directory also needs to be declared as resource in the POM file

<!-- cut -->
    <build>
        <resources>
            <!-- cut -->
            <resource>
                <!-- Creates config directory for external config --> 
                <directory>src/config</directory>
                <targetPath>../config</targetPath>
            </resource>
        </resources>
            <!-- cut -->

So now I have a minimum skeleton for deploying a small standalone HTTP service. Keep in mind to restrict the 8080 port by using AWS Security Group so that the 8080 port is only accessible to appropriate instances.

Regarding the deployment, I’m guessing there will be few seconds downtime between the shutdown and running new service. This can be even much longer if I put some long task in the shutdown listener. I’m still thinking how to do this more seamlessly. Ideas needed 😉

Another thing I can do is passing the custom JSON in the OpsWorks to the `application.properties` configuration file.

I can define something like this.

{
    "springboot-example": {
        "config": {
            "server.port" : 3333,
            "management.port": 3334
        }
    }
}

It’s a good idea to make the server port and the management port more configurable among other things. And I can write that JSON in the `application.properties` using the deploy hook `before_symlink.rb` and modify the `before_restart.rb` to use the custom JSON for the port numbers. This is very flexible since I can define the custom JSON in the Stack level and also in the deployment level.


Another snippets today.

s3cmd put \
    --add-header=Expires:"`date -R --date='10 years'`"; \
    --add-header=Cache-Control:'max-age=31536000, public' \
    -P \
    [file] [bucket]

Update

s3cmd put \
    --add-header=Expires:"`date -u +"%a, %d %b %Y %H:%M:%S GMT" --date='10 years'`"  
    --add-header=Cache-Control:'max-age=31536000, public'  \   
    -P \
    [file] [bucket]

As usual, I wrote this because I always forgot about this.

Lately I have installed some services in my development laptop such as `elasticsearch`. And some of these services got automatically started every time the laptop start.

So turns out I can just remove all of the symlinks of init scripts in the `/etc/rc?.d` directory.

More explanation can be read here.


Beberapa kali saya mendengar klaim CloudKilat tentang storagenya yang kompatibel dengan S3. Sebenarnya tidak terlalu heran karena banyak provider yang menyediakan storage yang kompatibel dengan AWS S3 ataupun Openstack Swift. Sebenarnya saya tidak sedang mencari storage baru di luar S3, tapi sedang penasaran saja.

Untuk paket harganya, KilatStorage ini menawarkan 3 jenis paket: 10GB @ USD 0.17/GB (IDR 20000/10GB), 50GB @ USD 0.156/GB (IDR 90000/50GB), dan 100GB @ USD 0.143/GB (IDR 165000/100GB). Kalau dibandingkan dengan AWS dan Rackspace yang berturut-turut menawarkan USD 0.095/GB dan USD 0.1/GB untuk 1TB pertama, memang KilatStorage bisa dibilang sedikit lebih mahal.

Model pembayarannya juga agak berbeda, berhubung di Indonesia sangat jarang sekali orang yang menggunakan kartu kredit, maka KilatStorage di sini menerima pembayaran menggunakan transfer.

CloudKilat Order

Biasanya kalau di AWS dia akan otomatis menarik dari kartu kredit, sementara di sini kita diharuskan transfer setiap bulan (saya tidak melihat pilihan pembayaran tahunan di sini). Karena kebetulan dia layanannya memang gak seelastis AWS (yang langsung memotong per gigabyte yang digunakan), jadinya pilihan transfer terlihat tidak terlalu merepotkan (meski kalau harus transfer setiap bulannya nampaknya agak repot jatuhnya). Berarti ya memang seperti hosting biasa, kalau exceed berarti tinggal return error saja.

Setelah melakukan pembayaran nanti api key sama secret keynya akan tampil di Client Area > My Services > View Details. Untuk endpointnya tidak diberitahu di email konfirmasi, tapi setelah saya komplain via twitter eh ternyata ada email pemberitahuan. KilatStorage ini endpointnya di kilatstorage.com. Tapi sepertinya akses api key

Screenshot from 2013-09-18 14:19:25

Berhubung KilatStorage ini kompatibel dengan AWS S3, saya mencoba mengetes menggunakan s3cmd. Ternyata bisa, hanya perlu mengubah beberapa konfigurasi yakni access_keyhost_base, host_bucket, secret_key.

host_base = kilatstorage.com
host_bucket = %(bucket)s.kilatstorage.com

Setelahnya sih mudah

petra@petra-laptop:~/Pictures$ s3cmd --config ~/.s3cfg-kilatstorage put -P cloudkilat-13.png s3://blog.petrabarus.net/images/2013/09/cloudkilat-13.png -> s3://blog.petrabarus.net/images/2013/09/cloudkilat-13.png [1 of 1]
32081 of 32081 100% in 0s 249.60 kB/s done
Public URL of the object is: http://blog.petrabarus.net.kilatstorage.com/images/2013/09/cloudkilat-13.png

(note: Untuk s3cmd versi 1.0.0, dia public URLnya tetap ‘s3.amazonaws.com’ bukan ‘kilatstorage.com’ karena ini.)

Seharusnya juga ini kompatibel dengan boto dan SDK AWS.

Untuk support HTTPS juga nampaknya storage ini belum ada certificate key untuk kilatstorage.com. Not really a big deal, though.

Satu lagi yang agak aneh adalah di bagian response headernya. Biasanya ketika mengupload public image, kita mengeset “Cache-Control” dan “Expires” untuk HTTP caching. Ketika saya coba,

petra@petra-laptop:~/Pictures$ s3cmd --config ~/.s3cfg-kilatstorage put -P cloudkilat-13.png --add-header "Cache-Control: max-age=31536000, public" --add-header "Expires: Sat, 13 Sep 2014 09:26:34 GMT" s3://blog.petrabarus.net/images/2013/09/cloudkilat-131.png
cloudkilat-13.png -> s3://blog.petrabarus.net/images/2013/09/cloudkilat-131.png  [1 of 1]
 32081 of 32081   100% in    0s    63.51 kB/s  done
Public URL of the object is: http://blog.petrabarus.net.kilatstorage.com/images/2013/09/cloudkilat-131.png

Hasilnya adalah seperti demikian.

HTTP/1.1 200 OK
Server: nginx/1.4.1
Date: Wed, 18 Sep 2013 07:41:11 GMT
Content-Type: image/png
Content-Length: 32081
Connection: keep-alive
ETag: "0205b4472385bb10ffbd0ad2dc34487c"
Cache-Control: max-age=31536000, public
Last-Modified: Wed, 18 Sep 2013 07:39:31 UTC
Front-End-Https: on
Cache-Control: public, must-revalidate
Strict-Transport-Security: max-age=2592000; includeSubdomains

Bisa dilihat di atas, header Cache-Control nya jadi double, dan header Expires nya tidak keubah.

Anyway, mungkin nanti bakal coba testing speednya untuk ngebandingin dengan AWS.


I was just looking a good way to mount /tmp directory on AWS EC2 ephemeral storage automatically. Then I stumbled on this post. Nice method if you are using Ubuntu (or other distribution that uses Upstart).

# File /etc/init/mounted-mnt.conf

# mounted-mnt - Binds /tmp to /mnt/tmp

description     "Binds /tmp to /mnt/tmp"

start on mounted MOUNTPOINT=/mnt

task

script
    test -d /mnt/tmp || mkdir -m 1777 /mnt/tmp
    mount --bind /mnt/tmp /tmp
end script

And the good idea behind this, as told in the post, is

Some servers, like Apache/Passenger, might create important temporary files on /tmp. Once rc.local – the last in the boot sequence – ran they would get hidden and confuse the servers.


Kemaren, seperti beberapa tahun sebelumnya, saya kembali mengisi sesi pengantar Linux untuk Pelatihan Nasional bagi Tim Olimpiade Komputer Indonesia yang akan mengikuti International Olympiads of Informatics tahun 2012 di Milan.

Pengantar Linux ini memiliki tujuan agar siswa bisa terbiasa untuk memrograman dalam lingkungan sistem operasi Linux. Ini tidak lain karena di IOI nanti lingkungan pemrograman yang disediakan adalah Linux. Dalam pengantar ini, selain memperkenalkan tentang Linux serta program-program apa saja yang ada di sana (Nautilus, GCC, GEdit, Geany, dsb), saya juga memperkenalkan bagaimana menggunakan terminal untuk membantu dalam pemrograman.

Materinya dapat dilihat di bawah. Semoga bermanfaat.

NB: Materi ini merupakan versi terbaru dari yang sudah saya tulis tahun lalu. Ada beberapa tambahan seperti time dan batch processing.


Sering kelupaan dan malas nyari-nyari lagi snippet di internet.

#!/bin/bash
dir='/path/to/dir';
for file in `ls $dir`; do
        echo $file;
done;


Ralat:

Tips dari Pak Jo

Kode di atas bisa tidak berjalan dengan baik untuk path yang memiliki karakter spasi. Berikut adalah kode yang sudah diralat.

#!/bin/sh
dir='/etc';
ls "$dir" | while read file ; do
	echo $file;
done;

Berikut adalah materi Pengenalan UNIX Command yang saya berikan pada Pelatihan Nasional I Tim Olimpiade Komputer Indonesia tahun lalu. Materi ini diberikan agar para peserta dapat terbiasa dengan lingkungan pemrograman pada sistem operasi *NIX dan memanfaatkan perintah-perintah shell untuk membantu pemrograman.

Semoga berguna

%d blogger menyukai ini: