Compare commits

...

10 commits

12 changed files with 681 additions and 14 deletions

View file

@ -39,14 +39,10 @@ disableAliases = true
identifier = "linkedin"
name = "Linkedin"
url = "https://www.linkedin.com/in/creekorful"
[[languages.en.menu.main]]
identifier = "500px"
name = "500px"
url = "https://500px.com/creekorful"
[[languages.en.menu.main]]
identifier = "website"
name = "Website"
url = "https://creekorful.dev"
url = "https://www.creekorful.org"
[[languages.en.menu.main]]
identifier = "source-code"
name = "Blog source code"

View file

@ -35,7 +35,7 @@ So, what was your favorite time this year?
# Resolutions
For those who remember I'm always trying to set some goals for the year. I've written a [blog post](https://blog.creekorful.com/2020/01/taking-new-year-resolutions-seriously/) last year
For those who remember I'm always trying to set some goals for the year. I've written a [blog post](https://blog.creekorful.org/2020/01/taking-new-year-resolutions-seriously/) last year
about this.
Out of the 4 big resolutions I have undertaken, **3** of them has been accomplished, so I'm pretty happy about it!

View file

@ -10,7 +10,7 @@ description = ""
showFullContent = false
+++
I have written an article on the provisioning of a Docker Swarm cluster from scratch ([you can read it here](https://blog.creekorful.com/how-to-provision-a-secure-docker-swarm-cluster-from-scratch)) and I have received a lot of comments stating that docker swarm is dead and that I should be moving to Kubernetes instead.
I have written an article on the provisioning of a Docker Swarm cluster from scratch and I have received a lot of comments stating that docker swarm is dead and that I should be moving to Kubernetes instead.
# What happened to docker?
@ -37,4 +37,4 @@ Apparently, a lot of docker entreprise customers has requested support and invol
That's a very exciting news: we will continue to have an easier Kubernetes alternative, an alternative that is better suited for simple workflow and test laboratories.
Happy hacking!
Happy hacking!

View file

@ -10,7 +10,7 @@ description = ""
showFullContent = false
+++
I used to manage a dozen VPS since many years: Zabbix, Gitlab/Gitlab CI, [private docker registry](https://blog.creekorful.com/2020/01/harbor-private-docker-registry/),
I used to manage a dozen VPS since many years: Zabbix, Gitlab/Gitlab CI, [private docker registry](https://blog.creekorful.org/2020/01/harbor-private-docker-registry/),
production environment (3 nodes docker swarm cluster), database server (MariaDB & MongoDB),
blog server (running [Ghost](https://ghost.org/)), logs collector ([Graylog](https://www.graylog.org/)), etc...

View file

@ -11,13 +11,13 @@ showFullContent = false
+++
This article is part of a series about Docker Swarm. For the first article please
check [here](https://blog.creekorful.com/how-to-install-traefik-2-docker-swarm/).
check [here](https://blog.creekorful.org/how-to-install-traefik-2-docker-swarm/).
On this short tutorial you'll learn how to deploy securely the Traefik built-in dashboard with HTTPS support and basic
authentication system.
This article assume that you have a working Docker Swarm cluster with Traefik running with HTTPS support. If not you can
following [this article](https://blog.creekorful.com/how-to-install-traefik-2-docker-swarm/) to get started.
following [this article](https://blog.creekorful.org/how-to-install-traefik-2-docker-swarm/) to get started.
------
@ -31,7 +31,7 @@ view configured entrypoints, existing routers, services, ...
## Enable the Dashboard and the API
Let's take the final docker compose file from
the [first tutorial](https://blog.creekorful.com/how-to-install-traefik-2-docker-swarm/) and add some instructions:
the [first tutorial](https://blog.creekorful.org/how-to-install-traefik-2-docker-swarm/) and add some instructions:
```yaml
version: '3'
@ -163,7 +163,7 @@ networks:
Configure the exposure of the Traefik dashboard on the **traefik-ui.local** domain name, using the websecure entrypoint
with the letsencryptresolver. If you want more information about how to configure these, just check
my [first blog post about Traefik](https://blog.creekorful.com/how-to-install-traefik-2-docker-swarm/).
my [first blog post about Traefik](https://blog.creekorful.org/how-to-install-traefik-2-docker-swarm/).
```yaml
- "traefik.http.routers.traefik.service=api@internal"

View file

@ -0,0 +1,211 @@
+++
title = "Laravel: beware of $touches"
date = "2021-11-12"
author = "Aloïs Micard"
authorTwitter = "" #do not include @
cover = ""
tags = ["Laravel", "PHP"]
keywords = ["Laravel", "PHP"]
description = "Every framework have their limits."
showFullContent = false
+++
I have been using Laravel professionally since almost 1year, and I must say: I'm very impressed with the framework.
Everything's run smoothly, there's a feature for *(almost everything)* you can think of, so you *(almost)* never need to
reinvent the wheel.
This is very advantageous since you only focus on building your product features by features and spend less time working
on technical stuff who are less business valuable.
# Everything is fine... until it's not.
---
Recently we have faced really weird MySQL error at work:
> SQLSTATE[HY000]: General error: 1390 Prepared statement contains too many placeholders
What does it mean? It is certainly obvious: *the prepared statement contains too many placeholders*.
## What are placeholders again?
Placeholder are using in SQL prepared statement as template that will be replaced by the values when the query is
executed. Example:
```sql
insert into users (username, email) values (?, ?);
```
The following query contains placeholder for username and email (identified by the '?'). When this query will be executed
the values will be replaced.
## So where's the issue?
Following the stacktrace, I've determined that the error happened when doing a `$model->save()` call. So let's analyze
the model to see if something looks off:
```php
namespace App\Models;
/**
* @property Collection<Role> $roles
*/
class User extends Model
{
protected $touches = ['roles'];
public function roles(): BelongsToMany
{
return $this->belongsToMany(Role::class);
}
}
```
As you can see the model as nothing special declared, except for a little thing: the usage of `$touches`.
### What is $touches again?
*(I apologize in advance, the example below is really poor, but I couldn't come up with something else)*
Sometime, it may be useful to bump the `updated_at` of a model:
Let's see you are building an application to monitor the uptime of a website. Each time the website has been checked
you'll certainly want to bump the updated_at column of the Website model in order to display the value on the
interface (like a last_checked feature).
How do you touch a model to bump updated_at? Well using `$model->touch()` of course!
Okay thanks but what's with `$touches`?
![](/img/wegettherewhenwegetthere.png)
---
The role of the `$touches` variable is being able to *touch* (bump updated_at) element of a child collection when saving the
parent.
If we take our previous `User` model as example: each time you'll call `$user->save()` it will touch the roles relation (as
defined in `$touches`). Since the relation is a belongs to many it will invoke the following code:
```php
namespace Illuminate\Database\Eloquent\Relations;
class BelongsToMany extends Relation
{
public function touch()
{
$key = $this->getRelated()->getKeyName();
$columns = [
$this->related->getUpdatedAtColumn() => $this->related->freshTimestampString(),
];
// If we actually have IDs for the relation, we will run the query to update all
// the related model's timestamps, to make sure these all reflect the changes
// to the parent models. This will help us keep any caching synced up here.
if (count($ids = $this->allRelatedIds()) > 0) {
$this->getRelated()->newQueryWithoutRelationships()->whereIn($key, $ids)->update($columns);
}
}
}
```
This query will basically generate a single update to bump the roles.updated_at column. Something like this:
```sql
update roles set roles.updated_at = now() where roles.id in (1, 2, 3)
```
Will be executed. (in this example the user has the role 1, 2 and 3 affected)
### And the problem?
Well, as you may see it coming the problem was... We have models with more than **120,000** child in their relationship.
And since Laravel is trying to execute the update request in one shot, it has encountered a MySQL limit: the placeholder
limit.
This limit in MySQL is currently at
65,535 ([see this MySQL commit](https://github.com/mysql/mysql-server/blob/3290a66c89eb1625a7058e0ef732432b6952b435/sql/sql_prepare.cc#L1505)).
## How to handle such case?
The way we have handled this situation was simply by not using `$touches`, and manually doing the touches chunk by chunk
on the roles to not reach the limit.
I have chosen to use [listeners](https://laravel.com/docs/8.x/events#registering-events-and-listeners) for that. The
idea behind listeners is really simple: each time a model is _created_, _updated_, _saved_, _deleted_, an event is
dispatched, and you can react on it by writing special listener.
### Define the UserSaved event
The first thing is to create an event that will be fired when the User model is saved.
```sh
php artisan make:event UserSaved
```
and then references it in the model:
```php
namespace App\Models;
/**
* @property Collection<Role> $roles
*/
class User extends Model
{
protected $dispatchesEvents = [
'saved' => UserSaved::class,
];
public function roles(): BelongsToMany
{
return $this->belongsToMany(Role::class);
}
}
```
### Define the UserListener
Then we'll need to create the listener that will handle the user events:
```php
php artisan make:listener UserListener
```
and then we'll need to listen for this particular event:
```php
namespace App\Listeners;
class UserListener
{
public function handleUserSaved(UserSaved $event)
{
// Take roles ids by batch of 1000 and run a single SQL query
// to bump updated_at.
$event->user->roles()->chunk(1000, function (Collection $role) {
Role::whereIn('id', $role->pluck('id'))->update(['updated_at' => Carbon::now()]);
});
}
}
```
## Isn't this workaround odd? Shouldn't $touches work out-of-the-box?
Eh, while this may be *opinionated* (but that's my blog :p). I guess yes. As a user of Laravel I would either expect:
- The framework to handle such cases
- Having a note somewhere in the docs that explain the limits of `$touches`
I have raised an [issue](https://github.com/laravel/framework/issues/39259) to laravel/framework to discuss this bug.
After a bit of discussion it has come up that fixing the framework may not be the best thing to do since this use-case
is quite rare and the fix is a bit opinionated.
Therefore, opening a [pull request](https://github.com/laravel/docs/pull/7373) in laravel/docs to mention the technical
limits of `$touches`
was the logical follow-up to do. Sadly, the PR was rejected without taking time to think about it.
I must say I'm a bit disappointed of how the situation has ended, but... *meh*.
Happy hacking!

View file

@ -0,0 +1,302 @@
+++
title = "Laravel dynamic SMTP mail configuration"
date = "2021-11-09"
author = "Aloïs Micard"
authorTwitter = "" #do not include @ cover = ""
tags = ["Laravel", "PHP", "Tutorial"]
keywords = ["Laravel", "SMTP", "Dynamic", "PHP", "Tutorial"]
description = "How to use dynamic SMTP credentials with Laravel."
showFullContent = false
+++
Hello friend...
It has been a while.
I have been very busy lately with work, open source and *life* that I didn't find the energy to write a blog post.
Despite having some good ideas, I wasn't really in the mood.
Hopefully, I now have the energy and the subject to make a good blog post: let's talk
about [Laravel](https://laravel.com/) and emails!
# 1. Laravel and SMTP
## 1.1. Configuration
Laravel SMTP [Mail](https://laravel.com/docs/8.x/mail) support is truly awesome and work out-of-the-box without
requiring anything more than a few env variables:
```text
MAIL_MAILER=smtp
MAIL_HOST=mail1.example.org
MAIL_PORT=587
MAIL_USERNAME=demo@example.org
MAIL_PASSWORD=foo
MAIL_ENCRYPTION=tls
MAIL_FROM_ADDRESS=no-reply@example.org
MAIL_FROM_NAME=Demo App
```
> Note: The initial setup only requires these environment variables because the smtp mailer is already configured by default in `config/mail.php`.
## 1.2. Creating an email
Once everything is configured, creating an email is as simple as running this command:
```shell
php artisan make:mail Greetings
```
This command will generate a sample email in `app/Mail/Greetings` and you'll just need to design it afterwards.
## 1.3. Sending an email
Sending an email to a user with Laravel can be either done
- using `\Illuminate\Notifications\RoutesNotifications::notify`:
```php
$user->notify(new Greetings());
```
- or using `\Illuminate\Support\Facades\Notification::send`:
```php
Notification::send($users, new Greetings());
```
> Note: the later syntax is especially useful when bulk sending emails.
See how simple it is? ~~I wonder what would be the limitations.~~
Eh! What if we need to 'whitelabelize' our application. :-)
---
# 2. Dynamic SMTP configuration
In our scenario, we have the need to [whitelabelize](https://en.wikipedia.org/wiki/White-label_product) our application:
each `User` will belongs to a `Provider` that will have custom SMTP settings. So when sending email to a user we need to
configure dynamically the mailer to use the SMTP credentials of his provider (`$user->provider`).
**Can Laravel help us doing so?**
After a bit of googling and reading the official documentation, there's no out-of-the-box support for dynamic SMTP
configuration, certainly because there would be 100x way of doing it, each way depending on your exact use-case.
**So, we're screwed?**
Not yet, because Laravel allows us to tweak almost **anything**, so we just need to find our way.
## 2.1. Designing the models
Here's a quick visualization of our models:
```php
namespace App\Models;
/**
* @property Provider $provider
*/
class User extends Model
{
public function provider(): BelongsTo
{
return $this->belongsTo(Provider::class);
}
}
```
For the `User` model there's nothing special: we only link the user to a `Provider`.
```php
namespace App\Models;
/**
* @property array $mail_configuration
* @property Provider $provider
*/
class Provider extends Model
{
protected $casts = [
'mail_configuration' => 'encrypted:array'
];
public function users(): HasMany
{
return $this->hasMany(User::class);
}
}
```
The `Provider` model has many `Users` and has a `mail_configuration` field which is encrypted and that will contain the
SMTP credentials.
The email configuration will be stored as an [encrypted](https://laravel.com/docs/8.x/encryption) JSON configuration. It
will look like this:
```json
{
"host": "smtp.example.org",
"port": 587,
"username": "foo",
"password": "bar",
"encryption": "tls",
"from_address": "no-reply@example.org",
"from_name": "Example"
}
```
## 2.2. Digging down the internals
Now that our models are ready, we must find a way to use the provider configuration to send the email. Let's dig down in
Laravel source code to understand how emails works:
Remember the two-ways of sending emails?
- `\Illuminate\Notifications\RoutesNotifications::notify`
- `\Illuminate\Support\Facades\Notification::send`
What we need to do is find the common path between these two methods, and see if we can override some
behavior in there.
![](/img/laravel-mail-internals.png)
As you can see here, the methods share the same execution path that end up
calling `\Illuminate\Notifications\Channels\MailChannel::send`. So how can we hook up into this path?
The answer lies in `\Illuminate\Notifications\NotificationSender::sendToNotifiable`:
```php
protected function sendToNotifiable($notifiable, $id, $notification, $channel)
{
if (! $notification->id) {
$notification->id = $id;
}
if (! $this->shouldSendNotification($notifiable, $notification, $channel)) {
return;
}
$response = $this->manager->driver($channel)->send($notifiable, $notification);
$this->events->dispatch(
new NotificationSent($notifiable, $notification, $channel, $response)
);
}
```
As you can see this method is looking for a driver to use with the `$channel` and finally calls the `send()`
method. So what we can do is registering a special SMTP driver that will use dynamic SMTP settings.
Fortunately registering a custom driver is a common use-case and there's a straightforward way.
## 2.3. Creating a custom MailChannel
```php
namespace App\Notifications\Channels;
class ProviderMailChannel extends MailChannel
{
public function send($notifiable, Notification $notification)
{
// TODO: override the SMTP configuration
parent::send($notifiable, $notification);
}
}
```
## 2.4. Registering the ProviderMailChannel
All we need to know is extend (i.e: register a custom driver creator) for the `mail` channel.
This way when an email is sent it will be sent using our `ProviderMailChannel`.
```php
namespace App\Providers;
class AppServiceProvider extends ServiceProvider
{
public function boot()
{
/** @var ChannelManager $channelManager */
$channelManager = $this->app->get(ChannelManager::class);
$channelManager->extend('mail', function (Application $application) {
return new ProviderMailChannel($application->get('mail.manager'), $application->get(Markdown::class));
});
}
}
```
## 2.5. Creating a custom Mailer
Now that we are hooked up into the mail sending flow, we need to actually send the email. For doing so we need to instantiate
custom `\Illuminate\Mail\Mailer` instance that will be configured using the provider settings. To register such dynamic
configurable service we will use the power of
the [Service container](https://laravel.com/docs/8.x/container#binding-basics).
```php
namespace App\Providers;
class AppServiceProvider extends ServiceProvider
{
public function register()
{
// Register a custom mailer named `custom.mailer` that will receive his configuration dynamically
$this->app->bind('custom.mailer', function ($app, $parameters) {
$transport = new Swift_SmtpTransport($parameters['host'], $parameters['port']);
$transport->setUsername($parameters['username']);
$transport->setPassword($parameters['password']);
$transport->setEncryption($parameters['encryption']);
$mailer = new Mailer('', $app->get('view'), new Swift_Mailer($transport), $app->get('events'));
$mailer->alwaysFrom($parameters['from_address'], $parameters['from_name']);
return $mailer;
});
}
public function boot()
{
/** @var ChannelManager $channelManager */
$channelManager = $this->app->get(ChannelManager::class);
$channelManager->extend('mail', function (Application $application) {
return new ProviderMailChannel($application->get('mail.manager'), $application->get(Markdown::class));
});
}
}
```
Now we can instantiate this custom Mailer by doing the following:
```php
$mailer = app()->make('custom.mailer', $configuration);
// do something with $mailer
```
Where `$configuration` is the custom SMTP configuration.
## 2.6. Plug the custom Mailer into the ProviderMailChannel
Finally, all we need to do is to read the `Provider` mail configuration
and use it to instantiate our `custom.mailer` and then, use it to send the actual email.
```php
namespace App\Notifications\Channels;
class ProviderMailChannel extends MailChannel
{
public function send($notifiable, Notification $notification)
{
$message = $notification->toMail($notifiable);
$mailer = app()->make('custom.mailer', $notifiable->provider->mail_configuration);
$message->send($mailer);
}
}
```
# 3. Conclusion
That all folks. You are now capable of sending email using dynamic SMTP credentials based on the use-case.
Happy hacking!

View file

@ -10,7 +10,7 @@ description = ""
showFullContent = false
+++
As you may already know, I have launched, with a Friend, [an Android application](https://blog.creekorful.com/pimp-your-phone-like-never-before/) to customize phone wallpapers randomly. The development of the app itself only took us 2 months and was quite fun. The release was really exciting and the first feedback from real users was encouraging. However, things didn't go as planned...
As you may already know, I have launched, with a Friend, [an Android application](https://blog.creekorful.org/pimp-your-phone-like-never-before/) to customize phone wallpapers randomly. The development of the app itself only took us 2 months and was quite fun. The release was really exciting and the first feedback from real users was encouraging. However, things didn't go as planned...
# Referral program failure

View file

@ -0,0 +1,154 @@
+++
title = "Parakeet: an IRC log renderer"
date = "2021-08-09"
author = "Aloïs Micard"
authorTwitter = "" #do not include @ cover = ""
tags = ["My Projects", "Golang"]
keywords = ["", ""]
description = "Generate beautiful HTML static files from IRC logs."
showFullContent = false
+++
I have started using a lot IRC lately since I became a [Debian Developer](https://wiki.debian.org/DebianDeveloper) (Debian
use a lot IRC and mailing lists to communicate).
For those who don't know what IRC is: it's basically the ancestor of modern chat messaging
like [Discord](https://discord.com/). IRC is a quite old piece of technology, coming from 1988.
One big drawbacks of IRC is that there's no such thing as a history, you need to be connected to receive the messages,
everything that happens when your offline will be missed.
Because of such limitation, people started using [Bouncer](https://en.wikipedia.org/wiki/BNC_(software)#IRC) or self-hosted
IRC client such as [The lounge](https://thelounge.chat/). These clients are always connected to the IRC channels in order
to allow the user to read the messages sent while he is away.
I'm personally using a self-hosted **The lounge** on my dedicated server. The lounge is storing the logs history on the
filesystem under `/var/opt/thelounge/logs/{username}/{server}/*.log` files.
The log files looks like this:
```
2021-03-29T13:16:36.853Z] *** creekorful (~creekorful@0002a854.user.oftc.net) joined
[2021-03-29T14:09:34.402Z] *** fvcr (~francisco@2002c280.user.oftc.net) quit (Server closed connection)
[2021-03-29T14:09:45.237Z] *** fvcr (~francisco@2002c280.user.oftc.net) joined
[2021-03-29T15:07:10.843Z] *** Maxi[m] (~m189934ma@00027b5d.user.oftc.net) quit (Server closed connection)
[2021-03-29T15:07:15.415Z] *** Maxi[m] (~m189934ma@00027b5d.user.oftc.net) joined
[2021-03-31T11:44:01.184Z] <b342> If a pipeline in my namespace failed to build, does it mean it that only me get the e-mail for the failed build or the project I forked from too ?
[2021-03-31T11:53:03.597Z] <Myon> should be only you
[2021-03-31T11:53:49.045Z] <Myon> plus I think it's only the person triggering the pipeline, not the whole project
[2021-03-31T12:22:27.901Z] *** sergiodj (~sergiodj@00014bc9.user.oftc.net) quit (Server closed connection)
[2021-03-31T12:23:42.039Z] *** sergiodj (~sergiodj@00014bc9.user.oftc.net) joined
[2021-03-31T14:35:02.708Z] <b342> Thanks Myon
```
I was wondering: *maybe a could render these log file into something more visual?*
# Parakeet
I built [Parakeet](https://github.com/creekorful/parakeet) (in Golang) to solve this issue.
Using it is a simple as: `./parakeet -input irc-log.log -output irc-log.html`
## How does it work?
### Parsing the log file
The tool first parse the provided log file, and extract the messages from it, while excluding all login / logout events.
```go
for {
line, err = rd.ReadString('\n')
if err != nil {
break
}
line = strings.TrimSuffix(line, "\n")
// Only keep 'message' line i.e which contains something like '] <username>'
if !strings.Contains(line, "] <") || !strings.Contains(line, ">") {
continue
}
// Approximate line parsing
date := line[1:strings.Index(line, "] <")]
username := line[strings.Index(line, "<")+1 : strings.Index(line, ">")]
content := line[strings.Index(line, "> ")+2:]
t, err := time.Parse(time.RFC3339, date)
if err != nil {
break
}
ch.Messages = append(ch.Messages, Message{
Time: t,
Sender: username,
Content: content,
})
}
```
### Rendering the messages
Thanks to the go [html/template](https://pkg.go.dev/html/template) package, it's fairly easy to generate HTML.
The template file looks like this:
```
<html lang="en">
<head>
<title>{{ .Name }}</title>
<meta charset="UTF-8">
</head>
<body>
<table>
{{ range .Messages }}
<div>
[{{ .Time.Format "2006-01-02 15:04:05" }}] {{ .Sender | colorUsername }} {{ .Content | applyUrl }}
</div>
{{ end }}
</table>
</body>
</html>
```
#### Assign color to the username
Here's a little trick I've done to assign a color to each username, making the log files easier to read.
```go
type Context struct {
Colors []string
Users map[string]string
}
func (c *Context) colorUsername(s string) template.HTML {
// check if username color is not yet applied
color, exist := c.Users[s]
if !exist {
// pick-up new random color for the username
color = c.Colors[rand.Intn(len(c.Colors)-1)]
c.Users[s] = color
}
return template.HTML(fmt.Sprintf("&lt;<span style=\"color: %s; font-weight: bold;\">%s</span>&gt;", color, s))
}
```
#### Make links clickable
And here's how I've made the links clickable, by making them HTML anchor.
I've used [github.com/mvdan/xurls](https://github.com/mvdan/xurls) to find the URLs.
```go
func (c *Context) applyURL(s string) template.HTML {
rxStrict := xurls.Strict()
return template.HTML(rxStrict.ReplaceAllStringFunc(s, func(s string) string {
return fmt.Sprintf("<a href=\"%s\">%s</a>", s, s)
}))
}
```
# Conclusion
I'm using this tool to mirror the logs for the channel I'm using.
The results are available [here](https://irc-logs.creekorful.org/) and updated daily.
Happy hacking!

View file

@ -0,0 +1,4 @@
This website is open source. <br/>
Feel free to submit a PR if you spot a mistake :)<br/><br/>
<a href="https://github.com/creekorful/blog">https://github.com/creekorful/blog</a>

Binary file not shown.

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 115 KiB