Skip navigation

Tag Archives: aws


This was just an idea popped up when I had trouble sleeping last night. Basically we can use Thrift for defining the data and service going in and out Lambda service instead of plain REST API.

Below is the code for the server. To make it easier to deploy I am using Serverless Framework.

This is still lack authentication process whatsoever so I don’t think this should be use in production until I figure it out.

The full source code is hosted here: https://github.com/petrabarus/aws-lambda-thrift-example


Just dropping by here


For quite some time I’ve been looking for a very easy way to deploy a small service using Java that is accessible through HTTP and self-hosted, or in other words, it doesn’t need a installed server. I was initially thinking about using embedded Tomcat or Jetty by writing the server code manually.

But then it turns out that there is this project from Spring built specifically for this kind of application: the Spring Boot. As quoted from the web,

Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that can you can “just run”.

By using this framework, I only need to focus on building the service rather than writing the HTTP server. And running the service is just as simple as using plain old `java -jar`.

The whole source code can be seen in this URLĀ https://github.com/petrabarus/springboot-opsworks-example.

I started by using the sample code from Spring Boot.

import org.springframework.boot.*;
import org.springframework.boot.autoconfigure.*;
import org.springframework.stereotype.*;
import org.springframework.web.bind.annotation.*;

@Controller
@EnableAutoConfiguration
public class SampleController {

    @RequestMapping("/")
    @ResponseBody
    String home() {
        return "Hello World!";
    }

    public static void main(String[] args) throws Exception {
        SpringApplication.run(SampleController.class, args);
    }
}

The skeleton code for the controller can not be much simpler than this.

And here’s the POM file.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.petrabarus.maleskoding</groupId>
    <artifactId>springboot-opsworks-example</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.1.6.RELEASE</version>
    </parent>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <!-- Sets to use Jetty. No particular reasons -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jetty</artifactId>
        </dependency>
        <dependency>
            <!-- For deployment ready -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
    </dependencies>
    <build>
        <resources>
            <resource>
                <!-- Creates directory for OpsWorks config -->
                <directory>src/deploy</directory>
                <targetPath>../deploy</targetPath>
            </resource>
            <resource>
                <!-- Creates config directory for external config --> 
                <directory>src/config</directory>
                <targetPath>../config</targetPath>
            </resource>
        </resources>
        <plugins>
            <plugin>
                <!-- Creates JAR -->
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <!-- Creates zip distributable -->
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptor>src/assembly/dep.xml</descriptor>
                </configuration>
            </plugin>
        </plugins>
    </build>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <maven.compiler.source>1.7</maven.compiler.source>
        <maven.compiler.target>1.7</maven.compiler.target>
    </properties>
</project>

The XML code below is the minimum configuration to run the service. (The spring-boot-starter-jetty is used to make the service runs using Jetty instead of the default embedded Tomcat).

<!-- cut -->
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.1.6.RELEASE</version>
    </parent>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <!-- Sets to use Jetty. No particular reasons -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jetty</artifactId>
        </dependency>

<!-- cut -->

To build the JAR, I use this plugin

<!-- cut -->
    <build>
        <!-- cut -->
        <plugins>
            <plugin>
                <!-- Creates JAR -->
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
<!-- cut -->

Building the JAR is simply just by executing

$mvn clean install

and the JAR will be in the `target` directory, e.g. target/springboot-opsworks-example-1.0-SNAPSHOT.jar I run it using the usual java -jar command

$target/springboot-opsworks-example-1.0-SNAPSHOT

I can see the result by using curl

$curl localhost:8080
Hello World!

Now it’s time to deploy it using OpsWorks. In this example I use `HTTP archive`, but it’s better to use `S3 archive` to provide better security since it can use IAM credentials.

Screenshot from 2014-09-20 00:02:21

The ZIP-ed file for deployment can be created using `spring-boot-maven-plugin`

<!-- cut -->
    <build>
        <!-- cut -->
        <plugins>
            <plugin>
                <!-- Creates zip distributable -->
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptor>src/assembly/dep.xml</descriptor>
                </configuration>
            </plugin>
<!-- cut -->

Here’s the assembly XML configuration

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2" 
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">
    <id>bin</id>
    <formats>
        <format>zip</format>
    </formats>
    <fileSets>
        <fileSet>
            <directory>${project.build.directory}</directory>
            <outputDirectory>/</outputDirectory>
            <includes>
                <include>*.jar</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>${project.build.directory}/config</directory>
            <outputDirectory>config</outputDirectory>
            <includes>
                <include>*.xml</include>
                <include>*.properties</include>
                <include>*.yml</include>
            </includes>
        </fileSet>
        <fileSet>
            <directory>${project.build.directory}/deploy</directory>
            <outputDirectory>deploy</outputDirectory>
        </fileSet>
    </fileSets>
</assembly>

Now I can have a very simple build and distribution script like below. This can be use for Jenkins. When a new change is pushed to the git repository, Jenkins will download the change, build, test the source, and then pack and upload the binary.

$mvn clean install assembly:single && s3cmd put -P target/*.zip s3://path/to/the/upload.zip

As we know, deploying in OpsWorks is just a matter of click.

Screenshot from 2014-09-20 00:02:07

Note that deploying the OpsWorks app doesn’t mean running the service automatically. For that I need to use a deploy hook. Chef provides a way to execute a ruby script on steps in the deployment. Using the `before_restart.rb` I can execute a task that happens after the app directory changes its link from the old directory to the new deployed directory.

The content of `before_restart.rb` script is in below

script "runjar" do
    interpreter "bash"
    user "root"
    cwd release_path
    code <<-EOH
        java -jar *.jar > /var/log/springbootext1/app.log 2>&1 &
    EOH
end

I added the file in the `src/deploy` directory and I declare it as a resource in the POM file.

    <build>
        <resources>
            <resource>
                <!-- Creates directory for OpsWorks deploy hook -->
                <directory>src/deploy</directory>
                <targetPath>../deploy</targetPath>
            </resource>

The next question is how to shutdown the running service and update the new JAR. Spring Boot provides a production-ready feature called the actuator. Here I can put a shutdown for the service that will turn off the running service. To add the actuator, I define the dependency in the POM file.

<!-- cut -->
        <dependency>
            <!-- For deployment ready -->
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
<!-- cut -->

And in the deploy hook, before the run service script I added new script

script "shutdown" do
    interpreter "bash"
    user "root"
    cwd release_path
    code <<-EOH
        curl -XPOST "http://localhost:8080/shutdown" > /var/log/springbootext1/app.log 2>&1
    EOH
end

That script send a POST request to the current running service to shut itself down. As default the shutdown actuator is not enabled by default. I still need to add new configuration. To do this, I added a new file `scr/config/application.properties` and I added this line.

endpoints.shutdown.enabled=true

Spring Boot has a very flexible way of configuring the application. The simplest way is to have `config/application.properties` file in the JAR directory. The application JAR will check what’s inside the file and override the configuration.

The deploy directory also needs to be declared as resource in the POM file

<!-- cut -->
    <build>
        <resources>
            <!-- cut -->
            <resource>
                <!-- Creates config directory for external config --> 
                <directory>src/config</directory>
                <targetPath>../config</targetPath>
            </resource>
        </resources>
            <!-- cut -->

So now I have a minimum skeleton for deploying a small standalone HTTP service. Keep in mind to restrict the 8080 port by using AWS Security Group so that the 8080 port is only accessible to appropriate instances.

Regarding the deployment, I’m guessing there will be few seconds downtime between the shutdown and running new service. This can be even much longer if I put some long task in the shutdown listener. I’m still thinking how to do this more seamlessly. Ideas needed šŸ˜‰

Another thing I can do is passing the custom JSON in the OpsWorks to the `application.properties` configuration file.

I can define something like this.

{
    "springboot-example": {
        "config": {
            "server.port" : 3333,
            "management.port": 3334
        }
    }
}

It’s a good idea to make the server port and the management port more configurable among other things. And I can write that JSON in the `application.properties` using the deploy hook `before_symlink.rb` and modify the `before_restart.rb` to use the custom JSON for the port numbers. This is very flexible since I can define the custom JSON in the Stack level and also in the deployment level.


Another snippets today.

s3cmd put \
    --add-header=Expires:"`date -R --date='10 years'`"; \
    --add-header=Cache-Control:'max-age=31536000, public' \
    -P \
    [file] [bucket]

Update

s3cmd put \
    --add-header=Expires:"`date -u +"%a, %d %b %Y %H:%M:%S GMT" --date='10 years'`"  
    --add-header=Cache-Control:'max-age=31536000, public'  \   
    -P \
    [file] [bucket]

I was just looking a good way to mount /tmp directory on AWS EC2 ephemeral storage automatically. Then I stumbled on this post. Nice method if you are using Ubuntu (or other distribution that uses Upstart).

# File /etc/init/mounted-mnt.conf

# mounted-mnt - Binds /tmp to /mnt/tmp

description     "Binds /tmp to /mnt/tmp"

start on mounted MOUNTPOINT=/mnt

task

script
    test -d /mnt/tmp || mkdir -m 1777 /mnt/tmp
    mount --bind /mnt/tmp /tmp
end script

And the good idea behind this, as told in the post, is

Some servers, like Apache/Passenger, might create important temporary files on /tmp. Once rc.local ā€“ the last in the boot sequence ā€“ ran they would get hidden and confuse the servers.


Pada postingan kali ini saya akan menuliskan bagaimana menggunakan AWS SDK for PHP version 2 pada Yii. Sebelum menggunakan yang versi 2, saya sudah menggunakan versi 1 dan versi kedua ini lebih mudah digunakan karena dia sudah mendukung pengaturan kode PHP yang sudah terstruktur sesuai dengan rekomendasi standar yang sedang tren yakni PSR. Oleh karena itu penggunaan SDK v2 ini dapat dilakukan dengan menggunakan Composer.

Cara instalasi program Composer untuk melakukan instalasiĀ libraryĀ AWS SDK, dapat dilihat di sini.

Seperti instalasi paketĀ libraryĀ ala Composer, pertama-tama kita membuat spesifikasi paket pada berkas “composer.json”. Untuk menambahkan AWS SDK, isi “composer.json” adalah sebagai berikut.

{
  "config": {
    "vendor-dir": "protected/composer/vendors"
  },
  "require": {
    "aws/aws-sdk-php": "2.3.*@dev"
  }
}

Sebenarnya konfigurasi Composer untuk instalasi SDK sudah cukup pada baris 5 sampai 7, tetapi saya pribadi lebih suka menambahkan konfigurasi untuk menyimpanĀ library dari Composer pada direktori “protected/composer/vendors” agar tidak mengotori direktori lain seperti “protected/extensions” dan “protected/vendors“.

Setelah menambahkan konfigurasi tersebut, kita harus menjalankan perintah instalasinya.

petra@pc$composer install
Loading composer repositories with package information
Installing dependencies
  - Installing symfony/event-dispatcher (v2.2.1)
    Loading from cache

  - Installing guzzle/guzzle (v3.5.0)
    Loading from cache

  - Installing aws/aws-sdk-php (dev-master a299b46)
    Cloning a299b46693878b5a7ffaa30e236fbefe9113b6e0

symfony/event-dispatcher suggests installing symfony/dependency-injection (2.2.*)
symfony/event-dispatcher suggests installing symfony/http-kernel (2.2.*)
aws/aws-sdk-php suggests installing ext-apc (Allows service description opcode caching, request and response caching, and credentials caching)
aws/aws-sdk-php suggests installing doctrine/cache (Adds support for caching of credentials and responses)
aws/aws-sdk-php suggests installing monolog/monolog (Adds support for logging HTTP requests and responses)
aws/aws-sdk-php suggests installing symfony/yaml (Eases the ability to write manifests for creating jobs in AWS Import/Export)
Writing lock file
Generating autoload files

Sebelum bisa menggunakanĀ composer,Ā kita harus melakukanĀ includeĀ untukĀ autoloadingĀ Composer.

require_once(dirname(__FILE__).'/protected/composer/vendors/autoload.php');
// change the following paths if necessary
$yii=dirname(__FILE__).'/yii/framework/yii.php';
$config=dirname(__FILE__).'/protected/config/main.php';

// remove the following lines when in production mode
defined('YII_DEBUG') or define('YII_DEBUG',true);
// specify how many levels of call stack should be shown in each log message
defined('YII_TRACE_LEVEL') or define('YII_TRACE_LEVEL',3);

require_once($yii);
Yii::createWebApplication($config)->run();

Lalu kode di bagian controllernya seperti demikian

class SiteController extends Controller
{

    public function actionIndex()
    {
        if (Yii::app()->request->getIsPostRequest()) {
            $file = CUploadedFile::getInstanceByName('file');
            /* @var $file CUploadedFile */
            if (isset($file)) {
                $s3 = \Aws\S3\S3Client::factory(array(
                        'key' => 'YourAWSKEY',
                        'secret' => 'YourAWSSecret',
                        'region' => \Aws\Common\Enum\Region::SINGAPORE,
                    ));
                /* @var $s3 Aws\S3\S3Client */
                try {
                    $result = $s3->putObject(array(
                        'Bucket' => 'bucketname',
                        'Key' => $file->name,
                        'SourceFile' => $file->tempName,
                        ));
                    /* @var $result \Guzzle\Service\Resource\Model */
                    $url = $result->get("ObjectURL");
                    Yii::app()->user->setFlash('Success', "Upload Success: " . CHtml::link($url, $url));
                } catch (\Aws\S3\Exception\S3Exception $exc) {
                    /* @var $exc \Aws\S3\Exception\S3Exception */
                    $message = $exc->getMessage();
                    Yii::app()->user->setFlash('Failed', "Upload Failed: {$message}");
                }
            }
        }
        $this->render('index');
    }

}

BagianĀ controller akan menanganiĀ requestĀ dariĀ upload form kemudian melakukan pengunggahan ke S3. Untuk pengunggahan kita memerlukan kelas S3Client dari AWS SDK. Pada instansiasinya kita perlu mengesetĀ keyĀ danĀ secret dari akun AWS yang dimiliki, sertaĀ regionĀ tempat penyimpanan.

Lalu proses pengunggahan sendiri akan dilakukan dengan menginvokasiĀ method putObject.Ā Obyek yang dikembalikan olehĀ methodĀ ini hanya berupa model yang berisiĀ return valueĀ dari invokasi HTTP request yang dilakukan oleh Guzzle. Untuk mengambil URL dariĀ fileĀ yang telah diunggah, kita ambil saja atribut ObjectURL pada model tersebut.

Bagian viewnya sangat sederhana.

<h1>Upload file</h1>

<?php if (Yii::app()->user->hasFlash('Success')): ?>
    <div class="flash-success">
        <?php echo Yii::app()->user->getFlash('Success'); ?>
    </div>
<?php endif; ?>
<?php if (Yii::app()->user->hasFlash('Failed')): ?>
    <div class="flash-error">
        <?php echo Yii::app()->user->getFlash('Success'); ?>
    </div>
<?php endif; ?>

<?php echo CHtml::beginForm('', 'post', array('enctype' => 'multipart/form-data')); ?>
<?php echo CHtml::fileField('file'); ?>
<?php echo CHtml::submitButton(); ?>
<?php echo CHtml::endForm(); ?>

Nanti hasilnya akan seperti demikian.

Tampilan Sebelum Pengunggahan

Tampilan Sebelum Pengunggahan

Setelah melakukan pengunggahan, dia akan menampilkan URL yang telah disimpan oleh setFlash.

Tampilan Setelah Pengunggahan

Tampilan Setelah Pengunggahan

Dan di AWS S3 console, sudah bisa dilihat di dalam bucketnya kalau filenya sudah diunggah.

Isi Bucket Setelah Pengunggahan

Isi Bucket Setelah Pengunggahan

AWS SDK for PHP version 2 ini jauh lebih enak digunakan daripada versi sebelumnya karena sudah memenuhi standar PSR sehingga bisa digunakan dengan Composer serta dengan mudah diintegrasikan pada aplikasi Yii. Selain S3, SDK ini juga menyediakan banyakĀ library untuk layanan AWS lainnya seperti EC2, DynamoDB, dll.

Kodenya dapat dilihat di link ini.

Failed

Belakangan ini saya harus mengolah beberapa data yang jumlahnya cukup besar, tidak terlalu besar tetapi juga tidak cukup kecil untuk dijalankan di laptop sendiri. Cara yang saya gunakan adalah MapReduce menggunakan Apache Hadoop. Berhubung data yang besar itu sudah terlanjur saya simpan di Amazon S3 maka secara intuitif pengolahan datanya baiknya dilakukan menggunakan Amazon Elastic MapReduce (yang merupakan salah satu alasan saya belajar kilat menggunakan Hadoop).

Dan menjalankan skrip MapReduce di AWS EMR ini sebenarnya tidak terlalu sulit. Kalau sudah bisa berhasil menjalankan contoh aplikasi WordCount bawaannya Hadoop, menjalankannya di EMR hanya sekedar melakukan konfigurasi saja. Tutorial instalasi Hadoop untuk local machine dan cara menjalankan program contoh WordCount dapat dilihat di halaman ini.

Read More »

%d blogger menyukai ini: