Wednesday, 13 October 2021

Tuesday, 12 October 2021

Kutt : Free Modern URL Shortener

cd /opt/webapps/kutt

sudo -u www-data pm2 startup

sudo env PATH=$PATH:/usr/bin pm2 startup systemd -u wildanm --hp /var/www

sudo -u www-data pm2 save

Referensi

  1. Kutt, a modern URL shortener with support for custom domains. Shorten URLs, manage your links and view the click rate statistics, https://github.com/openthinklabs/kutt
  2. pm2 startup script, https://pm2.keymetrics.io/docs/usage/startup/

Saturday, 9 October 2021

UVDesk : Open Source Helpdesk System

install extention imap 

install mailparse : pecl install mailparse


Setelah composer install, jika ada error : 

Executing script cache:clear [KO]

 [KO]

Script cache:clear returned with error code 1

!!

!!  In ArrayNode.php line 220:

!!

!!    The child node "key" at path "cspoo_swiftmailer_mailgun" must be configured

!!    .

!!

!!

!!

Script @auto-scripts was called via post-update-cmd


Ikuti langkah disini : https://github.com/tehplague/swiftmailer-mailgun-bundle/issues/74#issuecomment-428552701

 

Untuk proses update jangan menuliskan perintah composer update, tapi lakukan hal berikut : 

composer update uvdesk/core-framework
composer update uvdesk/support-center-bundle
composer update uvdesk/mailbox-component
composer update uvdesk/automation-bundle
composer update uvdesk/extension-framework

 php bin/console swiftmailer:email:send --to=to@gmail.com --from='from@gmail.com'' --subject="test from cli" --body=test --mailer=smtp.mail.com

smtp

Mailgun


composer require cspoo/swiftmailer-mailgun-bundle php-http/guzzle5-adapter
composer require guzzlehttp/psr7

Error : 

doctrine/inflector/lib/Doctrine/Common/Inflector/Inflector.php:90)"} []
[2021-10-13 08:15:51] php.CRITICAL: Call to undefined method cspoo\Swiftmailer\MailgunBundle\Service\MailgunTransport::setPassword() {"exception":"[object] (Symfony\\Component\\Debug\\Exception\\FatalThrowableError(code: 0): Call to undefined method cspoo\\Swiftmailer\\MailgunBundle\\Service\\MailgunTransport::setPassword() at /opt/webapps/uvdesk/vendor/uvdesk/core-framework/Services/EmailService.php:500)"} []

Solusi : 

Pada file vendor/cspoo/swiftmailer-mailgun-bundle/Service/MailgunTransport.php

tambahkan properties dan methods berikut : 

        /**
     * The password for authentication.
     *
     * @var string
     */
    private $password;

    /**
     * Set the password to authenticate with.
     *
     * @param string $password
     */
    public function setPassword($password)
    {
        $this->password = $password;
    }
    
    
    /**
     * Get the password to authenticate with.
     *
     * @return string
     */
    public function getPassword()
    {
        return $this->password;
    }    

Perubahan ini sudah diterapkan pada openthinklabs/swiftmailer-mailgun-bundle

Konfigurasi .env : 
MAILER_URL=smtp://smtp.mailgun.org:587?encryption=tls&username=ausername&password=apassword
###< symfony/swiftmailer-bundle ###
MAILGUN_DOMAIN="domain.com"
MAILGUN_API_KEY="domain_key"
MAILGUN_SENDER="info@domain.com"


Untuk mengetes mengirim email, gunakan perintah : 

php bin/console swiftmailer:email:send --from=<from email> --to=<to email> --subject="Foo" --body="Bar" 

MailBox


Contoh konfigurasi yang dapat digunakan adalah seperti ini: 

config/packages/uvdesk_mailbox.yaml


Untuk menarik data dari mailbox, dapat menggunakan perintah : 

php bin/console uvdesk:refresh-mailbox email-address@example.com

Buat cron job untuk mengeksekusi perintah ini secara rutin, misal setiap 15 menit sekali. 

Mac


Jika perintah ini dijalankan di Mac, maka kita akan mendapatkan error : 

....
 Mailbox refreshed successfully !!!
PHP Fatal error:  Uncaught ErrorException: Notice: Unknown: [ALERT] Filesystem notification initialization error -- contact your mail administrator (check for configuration errors with the FAM/Gamin library) in Unknown:0
Stack trace:
#0 [internal function]: Symfony\Component\Debug\ErrorHandler->handleError(8, 'Unknown: [ALERT...', 'Unknown', 0, NULL)
#1 {main}
  thrown in Unknown on line 0

Fatal error: Uncaught ErrorException: Notice: Unknown: [ALERT] Filesystem notification initialization error -- contact your mail administrator (check for configuration errors with the FAM/Gamin library) in Unknown:0
Stack trace:
#0 [internal function]: Symfony\Component\Debug\ErrorHandler->handleError(8, 'Unknown: [ALERT...', 'Unknown', 0, NULL)
#1 {main}
  thrown in Unknown on line 0

Referensi 

  1. composer update caused uvdesk broken, https://github.com/uvdesk/community-skeleton/issues/263
  2. SSL3_GET_SERVER_CERTIFICATE:certificate verify failed #544, https://github.com/swiftmailer/swiftmailer/issues/544
  3. How to Setup Mailbox : UVdesk Open Source Helpdesk, https://www.uvdesk.com/en/blog/setup-mailbox-uvdesk-open-source-helpdesk/
  4. UVdesk- Custom Field, https://webkul.com/blog/uvdesk-custom-fields/

Wednesday, 8 September 2021

Utilities untuk Zendesk

 Export Tickets

Jika berlangganan paket Suite Team, maka kita tidak akan mendapat akses ke fungsionalitas Report untuk mengexport data ticket-ticket yang masuk. Jika menggunakan custom view, yg bisa diexport hanya 2000 tiket. 

Ada beberapa solusi menggunakan API yg telah dibuat oleh komunitas. Dintaranya timpalac/zendeskexport [1], kelemahannya, hanya bisa meng-export maksimal 1000 tiket dalam satu requeat. 

Zenpy pun menawarkan solusi, salah satu nya adalah yg telah ditulis oleh Paul [2][3], tapi sepertinya API nya sudah kadaluarsa (?). Setelah mencoba memodifikasi script diatas, Alhamdulillah, akhirnya bisa menarik data tiket Zendesk : 

credentials = {
'email' : 'email',
'token' : 'token',
'subdomain': 'subdomain'
}

# Import the Zenpy Class
from zenpy import Zenpy
from django.utils import timezone
import requests
import pandas as pd
import datetime as dt
import json
import time

zenpy_client = Zenpy(**credentials)

def export_data(start_date = dt.datetime(2020, 1, 1, tzinfo=timezone.utc)):

tickets_json = '['
tickets_df = pd.DataFrame()
org_df = pd.DataFrame()
users_df = pd.DataFrame()
df_tickets = pd.DataFrame(columns=['id', 'requester_id', 'submitter_id',
'created_at', 'subject', 'description',
'tags', 'channel', 'status'])

result_generator = zenpy_client.tickets.incremental(start_time=start_date, include=['users','organizations','groups'])
for ticket in result_generator:
ticket = ticket.to_dict()
row = {
'id': ticket['id'],
'requester_id': ticket['requester_id'],
'submitter_id': ticket['submitter_id'],
'created_at': ticket['created_at'],
'subject': ticket['subject'],
'description': ticket['description'],
'tags': ticket['tags'],
'channel': ticket['via']['channel'],
'status': ticket['status'],
}

df_tickets = df_tickets.append(row, ignore_index=True)

print(str(len(df_tickets)) +" Tickets Imported")



df_tickets.to_csv('tickets_export_20200101_20210908.csv')


export_data()



Dari ekosistem Pentaho, ada yang menawarkan plugin untuk Zendesk [4], tapi ketika mencoba untuk dibuild, masih error : 

[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< org.pentaho.pdi:pdi-zendesk-plugin >-----------------
[INFO] Building PDI Zendesk Plugin 1.3.2-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  0.149 s
[INFO] Finished at: 2021-09-08T20:07:24+07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.pentaho.maven.plugins:license-helper-maven-plugin:1.26 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.pentaho.maven.plugins:license-helper-maven-plugin:jar:1.26: Could not transfer artifact org.pentaho.maven.plugins:license-helper-maven-plugin:pom:1.26 from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [pentaho-public (http://nexus.pentaho.org/content/groups/omni/, default, releases+snapshots)] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException

Referensi

  1. ZenDesk Export, https://github.com/timpalac/zendeskexport
  2. Zenpy, https://github.com/facetoe/zenpy
  3. How to Export all Tickets from Zendesk using the API and Python, https://www.pauldesalvo.com/how-to-export-all-tickets-from-zendesk-using-the-api-and-python/
  4. Zendesk Plugin for PDI, https://github.com/matthewtckr/pdi-zendesk-plugin

Monday, 12 July 2021

Goole Data Studio

Choropleth Map


Beberapa Provinsi yg belum masuk ke GeoChart : Papua Barat dan Maluku, tapi jika menggunakan code provinsi dari ISO_3166-2:ID, bisa, seperti dijelaskan di [4]

Referensi

Thursday, 29 April 2021

Social Network Analysis

Agar dapat dijalan dibackground dan tetap berjalan meskipun kita exit dari ssh, dapat menggunakan nohup : 

scrapy crawl TweetScraper -a query=TKA CHINA since:2020-05-10 until:2020-05-16

Data hasil scraping disimpan ke database [5], hanya untuk memprosesnya, untuk mempermudah, kita dapat menggunakan Pentaho Data Integration [6]. 

Jika ada masalah ketika memproses data ke database dengan Pentaho, bisa membaca [7][8]. 

Untuk analisa dan visualisasi dengan Gephi dapat membaca [10], hanya saja, dataset yg disebutkan disini, sudah tidak dapat diakses lagi. 

Jika ingin menggunakan Gephi - Twitter Streaming Importer dapat menonton [11].

Referensi

  1. TweetScraper,  https://github.com/jonbakerfish/TweetScraper
  2. Python Tutorial: Understanding Twitter JSON, https://www.youtube.com/watch?v=qrruEhHqwqY
  3. Text Mining, Networks and Visualization: Plebiscito Tweets, https://juanitorduz.github.io/text-mining-networks-and-visualization-plebiscito-tweets/
  4. Building a network graph from Twitter data, https://towardsdatascience.com/building-a-network-graph-from-twitter-data-a5e7b8672e3
  5. Building Twitter Conversation Graph, https://medium.com/analytics-vidhya/building-twitter-conversation-graph-5830bee5a7eb
  6. JSON-Input, https://pentaho.home.blog/2019/04/21/json-input/
  7. Incorrect string value: '\xF0\x9F\x8E\xB6\xF0\x9F…' MySQL , https://stackoverflow.com/questions/20411440/incorrect-string-value-xf0-x9f-x8e-xb6-xf0-x9f-mysql
  8. Emoji characters can't be written to Mysql database, https://www.programmersought.com/article/9501164118/
  9. Social Network Analysis and Visualization in Gephi, https://www.uu.nl/sites/default/files/clue_training_gephi.pdf
  10. Analyzing and visualizing your Twitter networks in Gephi , https://www.uu.nl/sites/default/files/analyzing_and_visualizing_your_twitter_networks_in_gephi.pdf
  11. Lab 11 Collect and Visualize #Twitter Social Network in #Gephi, https://www.youtube.com/watch?v=yYd8nSQRpis
  12. How to work with bigger datasets in Gephi, https://www.databulle.com/blog/random/gephi-ram.html
  13. Social Feed Manager, Helping researchers and archivists build social media collections, https://gwu-libraries.github.io/sfm-ui/

Tuesday, 2 February 2021

Stress Test dengan Locust

  Pertanyaan :

  1. Bagaimana cara melakukan stress test untuk TAO ? Login, kemudian klik delivery dan mengerjakan soal pertama dan seterusnya sampai end test. 
Real Browser support for Locust.io tidak mendukung Locust versi terbaru. Locust lebih untuk melakukan load test terhadap API end point. Mungkin bisa juga, mimicking real browser, ketika mengakses halaman, Locut mendownload semua asset nya, tapi tentu, tidak akan se-real, ketika menggunakan load test dgn real browser seperti WDS selenium.  

Referensi

  1. Locust, An open source load testing tool. https://locust.io/
  2. Real Browser support for Locust.io load testing, https://github.com/nickboucart/realbrowserlocusts
  3. extension-tao-oauth, Extension to easily configure an OAuth client for OAT platform., https://github.com/OpenThinkLabs/extension-tao-oauth
  4. Say Goodbye To Apache JMeter Drupal Load Testing with Locust.io, https://www.tag1consulting.com/blog/drupal-loadtest-locust

Sunday, 31 January 2021

Stress Test dengan JMeter

 Pertanyaan :

  1. Bagaimana cara melakukan stress test untuk TAO ? Login, kemudian klik delivery dan mengerjakan soal pertama dan seterusnya sampai end test. 

JMeter dan Selenium Webdriver 

Paduan JMeter dan Selenium Webdriver, setelah dicoba, sepertinya hanya cocok untuk keperluan functional testing, tidak cocok untuk load testing, karena 1 virtual user itu menggunakan 1 CPU, jadi jika ingin mensimulasikan, misal, 1000 concurrent user, agak berat [6]. 


Solusi alternatif yg ditawarkan, adalah menggunakan locust.io, tapi karena kita perlu testing, yang mendukung javascript, apakah  limitasi diatas masih berlaku ?

Referensi

  1. How to monitor End User Response Time of your Web Applications using JMeter + Webdriver, Containers, and Azure Pipelines. https://medium.com/microsoftazure/how-to-monitor-end-user-response-time-of-your-web-applications-using-jmeter-webdriver-4a609cbb4b57
  2. Web Driver Sampler, https://jmeter-plugins.org/wiki/WebDriverSampler/
  3. How to Fix Selenium’s “Element Is Not Clickable at Point”, https://www.testim.io/blog/selenium-element-is-not-clickable-at-point/
  4. Leverage your Load Testing using JMeter and Selenium WebDriver, https://www.ubik-ingenierie.com/blog/leverage-your-load-testing-using-jmeter-and-selenium-webdriver/
  5. Selenium API Docs, https://www.selenium.dev/selenium/docs/api/java/index.html?overview-summary.html
  6. How To Put Multiple User Loads Like ( 100 , 200 Users ) on a Premade Selenium Script In Python Webdriver In Odoo?, https://stackoverflow.com/questions/32437816/how-to-put-multiple-user-loads-like-100-200-users-on-a-premade-selenium-sc