Merge branch 'devel' into twilio

2025-03-14 10:05:07 +00:00 · 2024-12-09 15:13:45 +03:00
parent 0f91507e98 a5c55a1a3d
commit ce244ac5b1
7 changed files with 305 additions and 63 deletions
--- a/README_ko.md
+++ b/README_ko.md
@ -19,7 +19,7 @@ Tinode는 XMPP/ Jabber 가 아닙니다. Tinode는 XMPP와 호환되지 않습

 XMPP: XML에 기반한 메시지 지향 통신 프로토콜

-IM: Instant Messanger
+IM: Instant Messenger

 ## 설치 및 실행

--- a/docs/API.md
+++ b/docs/API.md
@ -1,66 +1,67 @@
-<!-- TOC depthFrom:1 depthTo:6 withLinks:1 updateOnSave:1 orderedList:0 -->
+<!-- TOC depthfrom:1 depthto:6 withlinks:true updateonsave:true orderedlist:false -->

 - [Server API](#server-api)
-	- [How it Works?](#how-it-works)
-	- [General Considerations](#general-considerations)
-	- [Connecting to the Server](#connecting-to-the-server)
-		- [gRPC](#grpc)
-		- [WebSocket](#websocket)
-		- [Long Polling](#long-polling)
-		- [Out of Band Large Files](#out-of-band-large-files)
-		- [Running Behind a Reverse Proxy](#running-behind-a-reverse-proxy)
-	- [Users](#users)
-		- [Authentication](#authentication)
-			- [Creating an Account](#creating-an-account)
-			- [Logging in](#logging-in)
-			- [Changing Authentication Parameters](#changing-authentication-parameters)
-			- [Resetting a Password, i.e. "Forgot Password"](#resetting-a-password-ie-forgot-password)
-		- [Suspending a User](#suspending-a-user)
-		- [Credential Validation](#credential-validation)
-		- [Access Control](#access-control)
-	- [Topics](#topics)
-		- [`me` Topic](#me-topic)
-		- [`fnd` and Tags: Finding Users and Topics](#fnd-and-tags-finding-users-and-topics)
-			- [Query Language](#query-language)
-			- [Incremental Updates to Queries](#incremental-updates-to-queries)
-			- [Query Rewrite](#query-rewrite)
-			- [Possible Use Cases](#possible-use-cases)
-		- [Peer to Peer Topics](#peer-to-peer-topics)
-		- [Group Topics](#group-topics)
-		- [`sys` Topic](#sys-topic)
-	- [Using Server-Issued Message IDs](#using-server-issued-message-ids)
-	- [User Agent and Presence Notifications](#user-agent-and-presence-notifications)
-	- [Trusted, Public, and Private Fields](#trusted-public-and-private-fields)
-		- [Trusted](#trusted)
-		- [Public](#public)
-		- [Private](#private)
-	- [Format of Content](#format-of-content)
-	- [Out-of-Band Handling of Large Files](#out-of-band-handling-of-large-files)
-		- [Uploading](#uploading)
-		- [Downloading](#downloading)
-	- [Push Notifications](#push-notifications)
-		- [Tinode Push Gateway](#tinode-push-gateway)
-		- [Google FCM](#google-fcm)
-		- [Stdout](#stdout)
-	- [Video Calls](#video-calls)
-	- [Messages](#messages)
-		- [Client to Server Messages](#client-to-server-messages)
-			- [`{hi}`](#hi)
-			- [`{acc}`](#acc)
-			- [`{login}`](#login)
-			- [`{sub}`](#sub)
-			- [`{leave}`](#leave)
-			- [`{pub}`](#pub)
-			- [`{get}`](#get)
-			- [`{set}`](#set)
-			- [`{del}`](#del)
-			- [`{note}`](#note)
-		- [Server to Client Messages](#server-to-client-messages)
-			- [`{data}`](#data)
-			- [`{ctrl}`](#ctrl)
-			- [`{meta}`](#meta)
-			- [`{pres}`](#pres)
-			- [`{info}`](#info)
+  - [How it Works?](#how-it-works)
+  - [General Considerations](#general-considerations)
+  - [Connecting to the Server](#connecting-to-the-server)
+    - [gRPC](#grpc)
+    - [WebSocket](#websocket)
+    - [Long Polling](#long-polling)
+    - [Out of Band Large Files](#out-of-band-large-files)
+    - [Running Behind a Reverse Proxy](#running-behind-a-reverse-proxy)
+  - [Users](#users)
+    - [Authentication](#authentication)
+      - [Creating an Account](#creating-an-account)
+      - [Logging in](#logging-in)
+      - [Changing Authentication Parameters](#changing-authentication-parameters)
+      - [Resetting a Password, i.e. "Forgot Password"](#resetting-a-password-ie-forgot-password)
+    - [Suspending a User](#suspending-a-user)
+    - [Credential Validation](#credential-validation)
+    - [Access Control](#access-control)
+  - [Topics](#topics)
+    - [me Topic](#me-topic)
+    - [fnd and Tags: Finding Users and Topics](#fnd-and-tags-finding-users-and-topics)
+      - [Query Language](#query-language)
+      - [Incremental Updates to Queries](#incremental-updates-to-queries)
+      - [Query Rewrite](#query-rewrite)
+      - [Possible Use Cases](#possible-use-cases)
+    - [Peer to Peer Topics](#peer-to-peer-topics)
+    - [Group Topics](#group-topics)
+    - [sys Topic](#sys-topic)
+  - [Using Server-Issued Message IDs](#using-server-issued-message-ids)
+  - [User Agent and Presence Notifications](#user-agent-and-presence-notifications)
+  - [Trusted, Public, and Private Fields](#trusted-public-and-private-fields)
+    - [Trusted](#trusted)
+    - [Public](#public)
+    - [Private](#private)
+  - [Format of Content](#format-of-content)
+  - [Out-of-Band Handling of Large Files](#out-of-band-handling-of-large-files)
+    - [Uploading](#uploading)
+    - [Downloading](#downloading)
+  - [Push Notifications](#push-notifications)
+    - [Tinode Push Gateway](#tinode-push-gateway)
+    - [Google FCM](#google-fcm)
+    - [Stdout](#stdout)
+  - [Video Calls](#video-calls)
+  - [Link Previews](#link-previews)
+  - [Messages](#messages)
+    - [Client to Server Messages](#client-to-server-messages)
+      - [{hi}](#hi)
+      - [{acc}](#acc)
+      - [{login}](#login)
+      - [{sub}](#sub)
+      - [{leave}](#leave)
+      - [{pub}](#pub)
+      - [{get}](#get)
+      - [{set}](#set)
+      - [{del}](#del)
+      - [{note}](#note)
+    - [Server to Client Messages](#server-to-client-messages)
+      - [{data}](#data)
+      - [{ctrl}](#ctrl)
+      - [{meta}](#meta)
+      - [{pres}](#pres)
+      - [{info}](#info)

 <!-- /TOC -->

@ -601,6 +602,21 @@ The `stdout` adapter is mostly useful for debugging and logging. It writes push

 [See separate document](call-establishment.md).

+## Link Previews
+
+Tinode provides an optional service which helps client applications generate link (URL) previews for inclusion into messages. The enpoint of this service (if enabled) is located at `/v0/urlpreview`. The service takes a single parameter `url`:
+
+```
+/v0/urlpreview?url=https%3A%2F%2Ftinode.co
+```
+The first several kilobytes of the document at the given URL is fetched by issuing an HTTP(S) GET request. If the returned document has content-type `text/html`, the HTML is parsed for page title, description, and image URL. The result is formatted as JSON and returned as
+
+```json
+{"title": "Page title", "description": "This is a page description", "image_url": "https://tinode.co/img/logo64x64.png"}
+```
+
+The link preview service requires authentication. It's exactly the same as authentication for [Out of Band Large Files](#out-of-band-handling-of-large-files).
+
 ## Messages

 A message is a logically associated set of data. Messages are passed as JSON-formatted UTF-8 text.
--- a/go.mod
+++ b/go.mod
@ -21,6 +21,7 @@ require (
 	github.com/tinode/snowflake v1.0.0
 	go.mongodb.org/mongo-driver v1.12.1
 	golang.org/x/crypto v0.21.0
+	golang.org/x/net v0.23.0
 	golang.org/x/oauth2 v0.16.0
 	golang.org/x/text v0.14.0
 	google.golang.org/api v0.148.0
@ -71,7 +72,6 @@ require (
 	github.com/xdg-go/stringprep v1.0.4 // indirect
 	github.com/youmark/pkcs8 v0.0.0-20201027041543-1326539a0a0a // indirect
 	go.opencensus.io v0.24.0 // indirect
-	golang.org/x/net v0.23.0 // indirect
 	golang.org/x/sync v0.4.0 // indirect
 	golang.org/x/sys v0.18.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
--- a/server/linkpreview.go
+++ b/server/linkpreview.go
@ -0,0 +1,207 @@
+package main
+
+import (
+	"encoding/json"
+	"errors"
+	"io"
+	"net"
+	"net/http"
+	"net/url"
+	"strings"
+	"time"
+	"unicode/utf8"
+
+	"golang.org/x/net/html"
+	"golang.org/x/net/html/atom"
+)
+
+type linkPreview struct {
+	Title       string `json:"title,omitempty"`
+	Description string `json:"description,omitempty"`
+	ImageURL    string `json:"image_url,omitempty"`
+}
+
+var client = &http.Client{
+	Timeout: time.Second * 2,
+	CheckRedirect: func(req *http.Request, via []*http.Request) error {
+		if err := validateURL(req.URL); err != nil {
+			return err
+		}
+		return nil
+	},
+}
+
+// previewLink handles the HTTP request, fetches the URL, and returns the link preview.
+// urlpreview?url=https%3A%2F%2Ftinode.co
+func previewLink(w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodGet && r.Method != http.MethodHead {
+		http.Error(w, http.StatusText(http.StatusMethodNotAllowed), http.StatusMethodNotAllowed)
+		return
+	}
+
+	// check authorization
+	uid, challenge, err := authHttpRequest(r)
+	if err != nil {
+		http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
+		return
+	}
+	if challenge != nil || uid.IsZero() {
+		http.Error(w, http.StatusText(http.StatusUnauthorized), http.StatusUnauthorized)
+		return
+	}
+
+	u := r.URL.Query().Get("url")
+	if u == "" {
+		http.Error(w, http.StatusText(http.StatusBadRequest), http.StatusBadRequest)
+		return
+	}
+
+	pu, err := url.Parse(u)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+	if err := validateURL(pu); err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	req, err := http.NewRequest(http.MethodGet, u, nil)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadRequest)
+		return
+	}
+
+	resp, err := client.Do(req)
+	if err != nil {
+		http.Error(w, err.Error(), http.StatusBadGateway)
+		return
+	}
+	defer resp.Body.Close()
+
+	if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices { // StatusCode != 20X
+		http.Error(w, "Non-OK HTTP status", http.StatusBadGateway)
+		return
+	}
+
+	body := http.MaxBytesReader(nil, resp.Body, 2*1024) // 2KB limit
+	if cc := resp.Header.Get("Cache-Control"); cc != "" {
+		w.Header().Set("Cache-Control", cc)
+	}
+
+	w.Header().Set("Content-Type", "application/json")
+	if r.Method == http.MethodHead {
+		w.WriteHeader(http.StatusOK)
+		return
+	}
+
+	if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
+		if err := json.NewEncoder(w).Encode(extractMetadata(body)); err != nil {
+			http.Error(w, "Failed to encode response", http.StatusInternalServerError)
+		}
+	}
+}
+
+func extractMetadata(body io.Reader) *linkPreview {
+	var preview linkPreview
+	var inTitleTag bool
+
+	tokenizer := html.NewTokenizer(body)
+lp:
+	for {
+		switch tokenizer.Next() {
+		case html.ErrorToken:
+			break lp
+
+		case html.StartTagToken, html.SelfClosingTagToken:
+			tag, hasAttr := tokenizer.TagName()
+			tagName := atom.Lookup(tag)
+			if tagName == atom.Meta && hasAttr {
+				var name, property, content string
+				for {
+					key, val, moreAttr := tokenizer.TagAttr()
+					switch atom.String(key) {
+					case "name":
+						name = string(val)
+					case "property":
+						property = string(val)
+					case "content":
+						content = string(val)
+					}
+					if !moreAttr {
+						break
+					}
+				}
+
+				if content != "" {
+					if strings.HasPrefix(property, "og:") {
+						switch property {
+						case "og:title":
+							preview.Title = content
+						case "og:description":
+							preview.Description = content
+						case "og:image":
+							preview.ImageURL = content
+						}
+					} else if name == "description" && preview.Description == "" {
+						preview.Description = content
+					}
+				}
+			} else if tagName == atom.Title {
+				inTitleTag = true
+			}
+
+		case html.TextToken:
+			if inTitleTag {
+				if preview.Title == "" {
+					preview.Title = tokenizer.Token().Data
+				}
+				inTitleTag = false
+			}
+
+		case html.EndTagToken:
+			inTitleTag = false
+		}
+		if preview.Title != "" && preview.Description != "" && preview.ImageURL != "" {
+			break
+		}
+	}
+
+	return sanitizePreview(preview)
+}
+
+func validateURL(u *url.URL) error {
+	if u.Scheme != "http" && u.Scheme != "https" {
+		return &url.Error{Op: "validate", Err: errors.New("invalid scheme")}
+	}
+
+	ips, err := net.LookupIP(u.Hostname())
+	if err != nil {
+		return &url.Error{Op: "validate", Err: errors.New("invalid host")}
+	}
+	for _, ip := range ips {
+		if ip.IsLoopback() || ip.IsPrivate() {
+			return &url.Error{Op: "validate", Err: errors.New("non routable IP address")}
+		}
+	}
+
+	return nil
+}
+
+func sanitizePreview(preview linkPreview) *linkPreview {
+	if utf8.RuneCountInString(preview.Title) > 80 {
+		preview.Title = string([]rune(preview.Title)[:80])
+	}
+	if utf8.RuneCountInString(preview.Description) > 256 {
+		preview.Description = string([]rune(preview.Description)[:256])
+	}
+	if len(preview.ImageURL) > 2000 {
+		preview.ImageURL = preview.ImageURL[:2000]
+	}
+
+	return &linkPreview{
+		Title:       strings.TrimSpace(preview.Title),
+		Description: strings.TrimSpace(preview.Description),
+		ImageURL:    strings.TrimSpace(preview.ImageURL),
+	}
+}
--- a/server/main.go
+++ b/server/main.go
@ -203,6 +203,9 @@ var globals struct {
 	// URL of the main endpoint.
 	// TODO: implement file-serving API for gRPC and remove this feature.
 	servingAt string
+
+	// Indicator if link preview generator is enabled.
+	linkPreviewEnabled bool
 }

 // Credential validator config.
@ -291,6 +294,11 @@ type configType struct {
 	// it's impossible to infer it.
 	DefaultCountryCode string `json:"default_country_code"`

+	// Enable service which generates link previews: in response to a GET request with a URL
+	// /v0/urlpreview?url=https%3A%2F%2Ftinode.co visit the URL, parse HTML, and return JSON like
+	// {"title": "Page title", description: "This is a demo page", image_url: "https://tinode.co/img/logo.png"}.
+	LinkPreviewEnabled bool `json:"link_preview"`
+
 	// Configs for subsystems
 	Cluster   json.RawMessage             `json:"cluster_config"`
 	Plugin    json.RawMessage             `json:"plugins"`
@ -729,6 +737,11 @@ func main() {
 		logs.Info.Println("Large media handling enabled", config.Media.UseHandler)
 	}

+	if config.LinkPreviewEnabled {
+		globals.linkPreviewEnabled = true
+		mux.HandleFunc(config.ApiPath+"v0/urlpreview", previewLink)
+	}
+
 	if staticMountPoint != "/" {
 		// Serve json-formatted 404 for all other URLs
 		mux.HandleFunc("/", serve404)
--- a/server/session.go
+++ b/server/session.go
@ -760,6 +760,7 @@ func (s *Session) hello(msg *ClientComMessage) {
 			"maxTagCount":        globals.maxTagCount,
 			"maxFileUploadSize":  globals.maxFileUploadSize,
 			"reqCred":            globals.validatorClientConfig,
+			"linkPreviewEnabled": globals.linkPreviewEnabled,
 		}
 		if len(globals.iceServers) > 0 {
 			params["iceServers"] = globals.iceServers
--- a/server/tinode.conf
+++ b/server/tinode.conf
@ -68,6 +68,11 @@
 	// If missing, the server will default to "US".
 	"default_country_code": "",

+	// Enable service which generates link previews: in response to a GET request with a URL
+	// /v0/urlpreview?url=https%3A%2F%2Ftinode.co visit the URL, parse HTML, and return JSON like
+	// {"title": "Page title", description: "This is a page description", image_url: "https://tinode.co/img/logo.png"}.
+	"link_preview_enabled": false,
+
 	// Large media/blob handlers: large files/images included in messages.
 	"media": {
 		// The name of the media handler to use.