{ "version": "https://jsonfeed.org/version/1", "title": "WalshyDev", "home_page_url": "https://walshy.dev", "feed_url": "https://walshy.dev/json", "description": "This is a personal site for me to list projects, write blog posts, etc. Don't take it too seriously.", "author": { "name": "Daniel Walsh", "url": "https://twitter.com/WalshyDev" }, "items": [ { "id": "404.md", "content_html": "
Oh noes. You seem to have lost your way. Click here to go back.
\nOrrrrrrrrrrrrr, just look at this cutie
\n\n(Credits to: https://www.reddit.com/user/mousito1/ - Reddit Post)
\n", "url": "https://walshy.dev/404", "title": "Not found", "summary": "Oh noes. You seem to have lost your way.", "date_modified": "2024-01-14T07:03:48.708Z" }, { "id": "donate.md", "content_html": "You can support me through:
\nHello, I am Walshy. I am a full-stack developer who mainly focuses on backend. I love to work on optimisations and analytics.\nI maintain many dashboards on my personal Grafana instance. This ranges from metrics on my projects, stats for my servers, UK COVID Vaccine stats, etc.
\nI am constantly working on many projects big or small and have a lot of notes. I decided I will try to categorise things a little more here.\nAlso, I wanted to write blogs but I like simple quick sites so, I am hosting them here using my own site generator.
\nClick here to see some of my projects.
\nClick here to see some of my blog posts.
Follow my socials:
\n\nContact me:
\nFollow my RSS feed:
\nSlimefun is a Minecraft plugin which aims to turn your Spigot Server into a modpack without ever installing a single mod. It offers everything you could possibly imagine. From Backpacks to Jetpacks! Slimefun lets every player decide on their own how much they want to dive into Magic or Tech.
\nSlimefun is an open source project with over 200 contributors, over 7000 commits and a Discord server with over 6,500 members!
\nStack:
\nLinks:
\n\nFirework is a very lightweight web framework for Java. Natively supports real ip headers, rate-limiting, middleware and more.
\nI've used Java to create REST APIs for a long time in and out of work, sadly no framework has fit my needs.\nI would like to not be creating a ton of util methods, wrapper classes, etc.
\nThis framework is being created so I (and hopefully others) can have a much better (and quicker!) experience creating REST APIs using Java.\nBuilt in Rate Limiting, Response Caching, middleware, reverse proxy support and a lot more! It's pretty hot 🔥
\nLinks:
\nA Discord bot made with JDA, FlareBot is a music and server administration bot along with a few other cool features.\nThis bot was created in September 2016 and was eventually discontinued in April 2018.
\nFlareBot grew to almost 50,000 servers which in 2018 made it one of the biggest bots on the platform! (These days, that's nothing, it's blown up a lot)
\nStack:
\nLinks:
\nSo, in 1.16.4 Pre Release 1 Mojang silently added a new option to server.properties
this was text-filtering-config
. Then I was curious about what it did. I set my sights on the code and dug in! I initially had a look at 1.16.5, I found the implementation (net.minecraft.server.network.TextFilterClient
) and had a look through it. I quickly realised though this wasn't finished and didn't currently function. In addition, it had no API endpoints yet which it was sending data to. So I decided to look at the snapshots. I diff'd between snapshots until I saw the endpoints get added. 21w07a worked on this implementation quite a bit and added the endpoints. I still diff'd up until 21w10a which at the time of writing is the latest snapshot.
Sadly it was now 02:30am, so I decided I'd sleep and look into this more tomorrow.
\nTomorrow came, and I dug straight back into the code, I made myself a config for the text filtering and set up a very simple express.js
server to capture the messages coming in, so I could confirm it's what I expected. I could also just have a nicer look at the data and headers.
So, the config is a JSON object with the following schema:
\n{\n "apiServer": String, // The server which to send filtering requests to\n "apiKey": String, // The API key to use (this is used in the Authorization header - Basic base64 auth)\n "ruleId": Integer, // This isn't really used for anything at the min\n "serverId": String, // The ID of this server, good for networks (this will likely be a UUID in the future if Mojang do a default implementation)\n "hashesToDrop": Integer, // The amount of "hashes" until the message is dropped. Keep reading for more info\n "maxConcurrentRequests": Integer // This is the thread pool size `Executors.newFixedThreadPool(maxConcurrentRequests)`\n}\n
\nFor an example of this config, check the Final Notes.
\nResult:
\n::ffff:127.0.0.1 POST /v1/chat\n{\n 'content-type': 'application/json; charset=utf-8',\n accept: 'application/json',\n authorization: 'Basic YWFhYWFhYWE=',\n 'user-agent': 'Minecraft server21w10a',\n 'cache-control': 'no-cache',\n pragma: 'no-cache',\n host: 'localhost:8000',\n connection: 'keep-alive',\n 'content-length': '138'\n}\n{\n rule: 1, // The rule ID you defined in server.properties\n server: 'test', // The server ID you defined in server.properties\n room: 'Chat', // Always chat, this will probably change for commands, books, etc\n player: 'f45cd935-7d4a-3527-9261-e4926d3ebb66', // The player's UUID\n player_display_name: 'HumanRightsAct', // The players username\n text: 'a' // The message the user sent\n}\n
\n(side note, hey Mojang, please add a space after "server" in the User-Agent, kthx <3)
\nI wasn't actually returning anything here, so I could see console complain that it wasn't getting a response. This is a good sign, it means they're also using the data I give back 😈
\nNow it's time to mess with the response! So, looking at the code I can see the response can have 3 values: response
, hashed
and hashes
. If the response
is false the message is sunk (not sent to any other users). If it is true, it then looks for a hashed
. If that isn't in the JSON it lets the message through. If it is there, it then goes to the hashes
Let's go through what we need to send back and what that JSON object should look like
\n{\n "response": Boolean, // If the message should be allowed - if false the message isn't sent to any users\n "hashed": String, // The modified message for the user to send (if it wasn't dropped or accepted)\n "hashes": String[] // The bad words/phrases that the user sent. This is used to see if the message should be dropped (hashesToDrop) \n}\n
\nSo, if I send {"response": false}
to all messages this would mean not a single one will go through! If I send {"response": true}
all messages go through.
But wait, does that mean I can only let messages through or sink them? Ah ha, no! The hashed
field defines the message which will be sent by the user (if it goes through). To test this I made my JSON response accept the message but send back a different string in hashed
.
{\n "response": true,\n "hashed": "I said a bad word",\n "hashes": []\n}\n
\nWe know hashes
is an array but don't want to test that yet so we'll make it empty. This response will make the player send I said a bad word
instead of what they originally sent. This means we can edit the message before it's sent to all players. Ever wanted to star out a bad word for players? Well now you can.
So, that's pretty simple right? If the message has a bad word you either respond with false and the message is sunk or you respond with true and modify it. But, what if you wanted to make it, so they'd need to say 2 bad words to be sunk? Well, that's where the hashes
field comes in to play. In the filtering config you define a hashesToDrop
this sets the bar for how many hashes
are needed in order to sink the message. So, if I set hashesToDrop
to 2 and then returned a JSON like this:
{\n "response": false,\n "hashed": "I said a bad word",\n "hashes": [\n "herobrine",\n "red"\n ]\n}\n
\nthen the message would be sunk because I specify 2 strings in hashes
and my hashesToDrop
config option is also set to 2. If I removed red
and only had 1 string in the hashes
array then the message would be replaced with "I said a bad word". This means you can filter out their bad word with *s or something like that. Pretty cool!
I have put a simple NodeJS script on Pastebin that you can use to test this yourself with here: https://pastebin.com/TjvzgYae
\nIf you write "herobrine" and/or "blobs" it will star that out of your message (or drop depending on your config).
\nThis was pretty fun to mess around with and definitely a nice feature, my only concern would be the privacy of this. You'd be sending every player join/leave, chat, book edit and probably more to Mojang (if they make themselves default). Now, the beauty is you can set up your own server like I did here. I can see people making a product out of this as well in the future. A web UI where you can set words and even regexs to match would be nice. Only Mojang know what the final result of this will be though! For now, there is no default config, so it isn't sent anywhere.
\nBelow I have put some notes on this. I hope I made it pretty clear what is happening and how.
\nHere is an example of it in action :)
\n\nEdit (2021-03-18): 21w11a didn't do any changes to text filtering
\nHere is my text-filtering-config
value (prettified):
{\n "apiServer": "http://localhost:8000",\n "apiKey": "aaaaaaaa",\n "ruleId": 1,\n "serverId": "test",\n "hashesToDrop": 2,\n "maxConcurrentRequests": 4\n}\n
\nHere's the minified you can post right into the server.properties file:
\n{"apiServer":"http://localhost:8000","apiKey":"aaaaaaaa","ruleId":1,"serverId":"test","hashesToDrop":2,"maxConcurrentRequests":4}\n
\nThe Authorization
header is just Basic <base64-apiKey>
. You can base64 decode mine to verify it matches my config here! I assume this will be more secure in the future.
API Endpoints:
\n/v1/chat
/v1/join
/v1/leave
If you return a 204 it will set the response as an empty JSON and use the default values. These currently are:
\nresponse
→ false
hashed
→ null
due to the response
being false
by default this will sink any message sent. Mojang, pls make this true
by default <3
There is code for book title and content however that is not currently sent
\n/tell
is not currently filtered so someone could say no no words there
You cannot modify colours as the text sent is just the string rather than the JSON Text Component (not tested if you can use the deprecated colour char)
\n", "url": "https://walshy.dev/blog/21_03_16-mc-new-text-filtering", "title": "MC's secret new text filtering", "summary": "Mojang are silently working on text filtering, find out all about it!", "date_modified": "2024-01-14T07:03:48.708Z" }, { "id": "21_04_01-why-is-java-11-string-repeat-so-quick.md", "content_html": "For a long time people have been making their own util methods for repeating a character or a sequence of characters. How would you do it? String concat? StringBuilder + for loop? Array fill? So does everyone else. With that in mind though, why is String#repeat() so much quicker? What is this sorcery? Well, in this blog post we'll dig into it!\nToday we'll be digging into single char repeating, it's the most common usage and my exact usage as to what prompted this blog post.
\nBefore we dig into the code of String.java
let's compare some timings:
\n\nNote: Timings done with jmh. Benchmark mode: Throughput, 2 Warmup Iterations and 1 Fork.
\n
The code:
\npublic class MyBenchmark {\n\n @Benchmark\n public void stringRepeat(Blackhole blackhole) {\n blackhole.consume("*".repeat(50));\n }\n\n // By the way, this is very bad. Don't do this <3\n @Benchmark\n public void repeatWithStringConcat(Blackhole blackhole) {\n String s = "";\n\n for (int i = 0; i < 50; i++) {\n s += "*";\n }\n\n blackhole.consume(s);\n }\n\n @Benchmark\n public void repeatWithStringBuilder(Blackhole blackhole) {\n final StringBuilder sb = new StringBuilder();\n\n for (int i = 0; i < 50; i++) {\n sb.append('*');\n }\n\n blackhole.consume(sb.toString());\n }\n\n @Benchmark\n public void repeatWithArraysFill(Blackhole blackhole) {\n final char c = '*';\n final char[] chars = new char[50];\n\n Arrays.fill(chars, c);\n\n blackhole.consume(new String(chars));\n }\n}\n
\nThe results:
\nBenchmark Mode Cnt Score Error Units\nMyBenchmark.repeatWithArraysFill thrpt 5 36391425.045 ± 1799053.802 ops/s\nMyBenchmark.repeatWithStringBuilder thrpt 5 14035510.440 ± 323791.700 ops/s\nMyBenchmark.repeatWithStringConcat thrpt 5 1969762.456 ± 49203.994 ops/s\nMyBenchmark.stringRepeat thrpt 5 73366664.415 ± 3612541.640 ops/s\n
\nAs we can see, the native String repeat is about twice as quick as Arrays.fill (which is the recommended way to do this pre-Java 9). What is this magic? Let's look together.
\nIf we take a look at the method, we see it's pretty simple:
\npublic String repeat(int count) {\n if (count < 0) {\n throw new IllegalArgumentException("count is negative: " + count);\n }\n if (count == 1) {\n return this;\n }\n final int len = value.length;\n if (len == 0 || count == 0) {\n return "";\n }\n if (Integer.MAX_VALUE / count < len) {\n throw new OutOfMemoryError("Required length exceeds implementation limit");\n }\n if (len == 1) {\n final byte[] single = new byte[count];\n Arrays.fill(single, value[0]);\n return new String(single, coder);\n }\n ...\n}\n
\nWe have a negative check, validation is important in a language or public library. Then if the count is 1 basically meaning don't repeat it, then it will just return itself. Simple 0 length check, if the String is empty or the count is 0 then just return an empty String. Checking the count vs length and throwing an error.
\nNow we are at the actual repeating part, since we just want to repeat a '*' that if (len == 1) {
block is for us. Let's take a closer look at that code:
final byte[] single = new byte[count];\nArrays.fill(single, value[0]);\nreturn new String(single, coder);\n
\nFirst, we make a byte array with the size of the supplied count. Next, we're filling the array with the first character in the string (which is a star). Lastly, we're making a String.\nPretty simple right? But wait... what is that coder
variable being passed to String?
Well, this is where the magic begins!
\nNormal implementations will be doing new String(array)
whereas the code here is doing new String(array, coder)
. That coder
variable is clearly speeding this up massively, so what is it?
/**\n * The identifier of the encoding used to encode the bytes in\n * {@code value}. The supported values in this implementation are\n *\n * LATIN1\n * UTF16\n *\n * @implNote This field is trusted by the VM, and is a subject to\n * constant folding if String instance is constant. Overwriting this\n * field after construction will cause problems.\n */\nprivate final byte coder;\n
\nSo, this byte
is an identifier for an encoding. Now let's look at where this is used:
String(byte[] value, byte coder) {\n this.value = value;\n this.coder = coder;\n}\n
\nFair enough, it's a package-private constructor which just sets the internal byte[] and coder
. What is the constructor that we mere peasants call?
String(char[] value, int off, int len, Void sig) {\n if (len == 0) {\n this.value = "".value;\n this.coder = "".coder;\n return;\n }\n if (COMPACT_STRINGS) {\n byte[] val = StringUTF16.compress(value, off, len);\n if (val != null) {\n this.value = val;\n this.coder = LATIN1;\n return;\n }\n }\n this.coder = UTF16;\n this.value = StringUTF16.toBytes(value, off, len);\n}\n
\nThe String(char[])
constructor calls this, another package-private one. We can see that coder
is being set to either LATIN1 (0) or UTF16 (1). We can also see that if COMPACT_STRINGS
is true then this will compress a UTF-16 string into LATIN1. I won't go into the compress code but if you want to see it then you can here.
It makes perfect sense. They already have the character as a byte, they fill the byte array and skip the compression. Meanwhile, we need to do this compression as we have a UTF-16 String.
\nIf you're interested in what that COMPACT_STRINGS
is then we can have a quick look but I won't go too deep into it and I recommend you do your own research. See the JEP.
They were added in Java 9 and if we take a look at the code there is a very nice comment explaining if this is disabled Strings are always encoded with UTF-16 (this is true pre-Java 9!). So, basically, if true it will encode String internally with Latin-1 rather than UTF-16. This provides a benefit for heap space as it's using an 8-bit character set.
\nMy question then was, when is this ever false? Well, there is a JVM argument to disable this exact thing, -XX:-CompactStrings
. Otherwise, the only time it's disabled is if the used characters cannot simply be encoded with Latin-1, in that case it will use UTF-16.
Since Java 9 implemented Latin-1 compression in String new Strings will need to be compressed to use this encoding, this adds additional overhead which cannot be avoided by the normal user but can in Java's own code. They instead use their coder
variable which signals the encoding of the current String. They just pass that instead of compressing or decoding.
If you're wondering what the timings are in Java 8, here they are:
\nBenchmark Mode Cnt Score Error Units\nMyBenchmark.repeatWithArraysFill thrpt 5 41988725.789 ± 732901.857 ops/s\nMyBenchmark.repeatWithStringBuilder thrpt 5 7905583.881 ± 92477.961 ops/s\nMyBenchmark.repeatWithStringConcat thrpt 5 2156015.552 ± 107153.092 ops/s\n
\nArrays.fill does seem to be on-par or slightly quicker than native 👀 but obviously, this suffers from more memory hungry strings.
\nOverall, if you're using Java 11 (current LTS) or any above 9, use String.repeat, if you're on Java 8, use Arrays.fill.
\n", "url": "https://walshy.dev/blog/21_04_01-why-is-java-11-string-repeat-so-quick", "title": "Why is Java 11's String repeat SO QUICK", "summary": "Let's dig into why Java 11's String#repeat method is so quick compared to normal util methods!", "date_modified": "2024-01-14T07:03:48.708Z" }, { "id": "21_09_10-handling-file-uploads-with-cloudflare-workers.md", "content_html": "Note:
\n\n\nThis tutorial won't be going into how Workers work or what they are exactly, I plan to have a blog post on this in the future.
\n
If you want to expand on this then you may also want a bucket (A B2/S3/other bucket not KFC bucket... though if you've got KFC gimme!)
\nFirstly, we need to set up the project. For this we will need the wrangler.toml
(project file for Wrangler), package.json
and the JS file we'll work on (I usually make this index.js
)
We can run the following commands to set this up quickly:
\n$ npm init\n$ wrangler init\n$ touch index.js\n
\nWe just need to edit the wrangler.toml
to have our account_id
, zone_id
and our route
. You can find your account ID and zone ID in the zone dashboard on the side.
In the end it should look like this:
\nname = "file-upload-tutorial"\ntype = "javascript"\naccount_id = "4e599df4216133509abaac54b109a647"\nzone_id = "8d0c8239f88f98a8cb82ec7bb29b8556"\nroute = "example.com/image-upload"\ncompatibility_flags = []\nworkers_dev = false\n
\nNow we have set the project up we're on to the key ingredient. You may have found old posts relating to this and seen that there was no native File API. This meant it was handled through strings and just wasn't a great experience. Well, not any more!
\nCompatibility flags (Released 30th July 2021 - Wrangler v1.19.0) allow for the parser to follow browser spec. So, to do this we will want to add compatibility_flags
and compatibility_date
to wrangler.toml
. We will add the flag formdata_parser_supports_files
which as the name indicates allows the parser to support files! For the compatibility_date
we will just point to today (formatted in "international standard" or ISO 8601). Our wrangler.toml
should now look like this:
name = "file-upload-tutorial"\ntype = "javascript"\naccount_id = "4e599df4216133509abaac54b109a647"\nzone_id = "8d0c8239f88f98a8cb82ec7bb29b8556"\nroute = "example.com/image-upload"\n# The needed parts for compatibility\ncompatibility_flags = [ "formdata_parser_supports_files" ]\n# This date should be set to today while you're developing.\n# This may cause a different runtime so best to only change\n# when you're developing and confirm it works.\ncompatibility_date = "2021-09-09"\nworkers_dev = false\n
\nNow we go on to the actual code part. When a user uploads an image this will be done as part of form data. This means that we want to parse the form data being received by the Worker so that we can handle the file. From this FormData we will get a File, with this we can get the MIME type, name, size and of course, the contents!
\nAssuming you already have the skeleton code we will start by parsing the form data that has been sent. This can be done with Request#formData like so (remember to await this!):
\nasync function handleRequest(request) {\n const formData = await request.formData();\n}\n
\nNow we have a FormData we need to get the File. We can do this by fetching the specific entry from FormData, I'm going to use the key file
but it could be anything so make sure this points to the key you're using (for an input
element this will be the name
attribute). Anyway, we can fetch it like so:
async function handleRequest(request) {\n const formData = await request.formData();\n const file = formData.get('file');\n}\n
\nAnd that's it! We now have a File! So, let's just test this, and you can build something with it.
\nTo test let's just print out a JSON with the name, type, size and a SHA-1 hash of the file. We can do this like so:
\naddEventListener('fetch', event => {\n event.respondWith(handleRequest(event.request))\n})\n\nasync function handleRequest(request) {\n // Parse the request to FormData\n const formData = await request.formData();\n // Get the File from the form. Key for the file is 'image' for me\n const file = formData.get('file');\n\n const hash = await sha1(file);\n\n return new Response(JSON.stringify({\n name: file.name,\n type: file.type,\n size: file.size,\n hash,\n }));\n}\n\nasync function sha1(file) {\n const fileData = await file.arrayBuffer();\n const digest = await crypto.subtle.digest('SHA-1', fileData);\n const array = Array.from(new Uint8Array(digest));\n const sha1 = array.map(b => b.toString(16).padStart(2, '0')).join('')\n return sha1;\n}\n
\nAnd let's test with a cURL (obviously use your worker URL here)
\n$ curl -X POST -F 'file=@/home/user/images/example.png' https://worker-name.example.workers.dev\n{"name":"example.png","type":"image/png","size":30283,"hash":"17946ec18d7b80f31e545acbc8baeb6294e39adc"}\n
\nAwesome, it works! :)
\nNow I got the image uploading working I wanted to build a B2 image uploader. I won't go through the whole process of that, but it's pretty simple.
\nb2_authorize_account
to get the auth details and then call b2_get_upload_url
then store the auth and upload details in KVcrypto.randomUUID()
) to make the file nameYou can find my code for this here: https://pastebin.com/0gnxKwQf
\nAnd see me testing it here:
\n$ curl -X POST -F 'image=@/home/walshy/images/resolved.png' https://file-upload-tutorial.walshy.workers.dev\n{"message": "Uploaded!", "file": "87bf9684-5d01-4d98-b9b6-f6a038661b4a.png", "b2Url": "https://f002.backblazeb2.com/file/worker-file-upload/87bf9684-5d01-4d98-b9b6-f6a038661b4a.png"}\n
\nand to prove it works, I will embed it here, and you can visit the URL yourself: https://f002.backblazeb2.com/file/worker-file-upload/87bf9684-5d01-4d98-b9b6-f6a038661b4a.png
\nHere are all my blog posts, feel free to tweet (@WalshyDev) or email (walshy@hey.com) any feedback to me!
\nSupport me by using my referral links!
\n