背景简介
日志脱敏 是常见的安全需求，最近公司也需要将这一块内容进行推进。看了一圈网上的案例，很少有既轻量又好用的轮子可以让我直接使用。我一直是反对过度设计的，而同样我认为轮子就应该是可以让人拿去直接用的。所以我准备分享两篇博客分别实现两种日志脱敏方案。
方案分析
-
logback MessageConverter + 正则匹配
上一篇介绍此方法- 优势
- 侵入性低、工作量极少， 只需要修改xml配置文件，适合老项目
- 劣势
- 效率低，会对每一行日志都进行正则匹配检查，效率受日志长度影响，日志越长效率越低，影响日志吞吐量
- 因基于正则匹配 存在错杀风险，部分内容难以准确识别
- 优势
-
fastjson Filter + 注解 + 工具类
本篇博客介绍- 优势
- 性能损耗低、效率高、扩展性强，精准脱敏，适合QPS较高日志吞吐量较大的项目。
- 劣势
- 侵入性较高，需对所有可能的情况进行脱敏判断
- 存在漏杀风险，全靠开发控制
- 优势
本篇博客部分代码未贴出，可以在上一篇博客中找到 传送门：Java日志脱敏——基于logback MessageConverter实现
fastjson Filter + 注解 + 工具类
流程图解
依托于 alibaba fastjson 提供的扩展能力，自定义ContextValueFilter，在将对象JSON序列化时，返回脱敏后的value，实现打印日志脱敏
代码案例
定义注解
定义 元注解 用于标记脱敏策略注解
package com.zhibo.log.sensitive.annotation.metadata;import java.lang.annotation.*;/*** @Author: Zhibo* @Description: 用于自定义 sensitive 脱敏策略注解，*/
@Inherited
@Documented
@Target(ElementType.ANNOTATION_TYPE)
@Retention(RetentionPolicy.RUNTIME)
public @interface SensitiveStrategy {
}
定义 手机号 脱敏策略注解
package com.zhibo.log.sensitive.annotation.strategy;import com.zhibo.log.sensitive.annotation.metadata.SensitiveStrategy;
import java.lang.annotation.*;/*** @Author: Zhibo* @Description: 手机号脱敏注解*/
@Inherited
@Documented
@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
@SensitiveStrategy
public @interface SensitiveStrategyPhone {
}
定义 身份证号码 脱敏策略注解
package com.zhibo.log.sensitive.annotation.strategy;import com.zhibo.log.sensitive.annotation.metadata.SensitiveStrategy;
import java.lang.annotation.*;/*** @Author: zhibo* @Description: 中国身份证号脱敏注解*/
@Inherited
@Documented
@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
@SensitiveStrategy
public @interface SensitiveStrategyIdNo {
}
定义 Map 类型扩展注解，考虑到部分自定义Bean 中会有一些Map的成员变量，而Map中可能也有敏感信息需要处理。
package com.zhibo.log.sensitive.annotation;import java.lang.annotation.*;/*** @Author: Zhibo* @Description: 针对Object对象中如果存在 Map参数，而Map存在敏感字段时使用<br></>* 如果对象中属性为一个Map，则可以使用这个注解指定，Map中特定Key的加密规则。*/
@Inherited
@Documented
@Target(ElementType.FIELD)
@Retention(RetentionPolicy.RUNTIME)
public @interface SensitiveMap {/*** 用于构建一个Map 传递格式 "key1","value1","key2","value2"....* key 为Map中需要脱敏的key，value为脱敏规则** 案例Demo：* @SensitiveMap({"phone", LogSensitiveConstants.STRATEGY_PHONE})* private Map<String,Object> map = new HashMap<>();* {* map.put("key", "value");* map.put("name","王大锤");* map.put("phone","18123456789");* }*/String[] value();}将注解与对应的脱敏策略方法绑定
package com.zhibo.log.sensitive.core.util.strategy;import com.zhibo.log.sensitive.annotation.metadata.SensitiveStrategy;
import com.zhibo.log.sensitive.annotation.strategy.*;
import com.zhibo.log.sensitive.api.IStrategy;
import com.zhibo.log.format.LogSensitiveConstants;
import com.zhibo.log.sensitive.core.strategory.*;import java.lang.annotation.Annotation;
import java.util.HashMap;
import java.util.Map;/*** @Author: zhibo* @Description: 系统中内置的策略映射、注解和实现之间映射*/
public final class SensitiveStrategyBuiltInUtil {private SensitiveStrategyBuiltInUtil(){}/** 注解和实现策略的映射关系 */private static final Map<Class<? extends Annotation>, IStrategy> CLASS_MAP = new HashMap<>();private static final Map<String, IStrategy> STRATEGY_MAP = new HashMap<>();static {StrategyAddress strategyAddress =new StrategyAddress();StrategyIdNo strategyIdNo = new StrategyIdNo();StrategyPhone strategyPhone = new StrategyPhone();CLASS_MAP.put(SensitiveStrategyAddress.class, strategyAddress);CLASS_MAP.put(SensitiveStrategyIdNo.class, strategyIdNo);CLASS_MAP.put(SensitiveStrategyPhone.class, strategyPhone);STRATEGY_MAP.put(LogSensitiveConstants.STRATEGY_ADDRESS, strategyAddress);STRATEGY_MAP.put(LogSensitiveConstants.STRATEGY_ID_NO, strategyIdNo);STRATEGY_MAP.put(LogSensitiveConstants.STRATEGY_PHONE, strategyPhone);}public static IStrategy getStrategy(String key){return STRATEGY_MAP.get(key);}/*** 获取对应的系统内置实现* @param annotationClass 注解实现类* @return 对应的实现方式*/public static IStrategy require(final Class<? extends Annotation> annotationClass) {return CLASS_MAP.get(annotationClass);}/*** 获取策略* @param annotations 字段对应注解* @return 策略*/public static IStrategy getStrategy(final Annotation[] annotations) {for (Annotation annotation : annotations) {SensitiveStrategy sensitiveStrategy = annotation.annotationType().getAnnotation(SensitiveStrategy.class);if (null != sensitiveStrategy) {return SensitiveStrategyBuiltInUtil.require(annotation.annotationType());}}return null;}
}
实现JSON序列化过滤器
关键代码了，基本的逻辑都在这里
package com.zhibo.log.sensitive.core.support.filter;import com.alibaba.fastjson.serializer.BeanContext;
import com.alibaba.fastjson.serializer.ContextValueFilter;
import com.zhibo.log.sensitive.annotation.SensitiveMap;
import com.zhibo.log.sensitive.api.IStrategy;
import com.zhibo.log.sensitive.core.context.SensitiveContext;
import com.zhibo.log.sensitive.core.util.strategy.SensitiveStrategyBuiltInUtil;import java.lang.annotation.Annotation;
import java.lang.reflect.Array;
import java.lang.reflect.Field;
import java.util.*;/*** @Author: Zhibo* @Description: 默认的上下文过滤器，支持处理 对象、数组、集合、Map 等类型数据，支持自定义脱敏策略*/
public class SensitiveContextValueFilter implements ContextValueFilter {/** 脱敏上下文 */private final SensitiveContext sensitiveContext;public SensitiveContextValueFilter(SensitiveContext context) {this.sensitiveContext = context;}@Overridepublic Object process(BeanContext context, Object object, String name, Object value) {// 对象为 MAP 的时候，FastJson map 对应的 context 为 NULLif(null == context) {//对象为MAP则检测是否存在指定脱敏策略if (null == sensitiveContext.getMapDesStrategy() || value == null){return value;}else {//执行匹配指定脱敏策略return desMapValue(name,value);}}// 信息初始化final Field field = context.getField();if (field == null){return value;}return handleSensitive(value,field);}/*** Map 类型数据脱敏* @param name key* @param value 未被脱敏的原对象* @return 脱敏后的新对象*/private Object desMapValue(String name, Object value){String desStrategy = sensitiveContext.getMapDesStrategy().get(name);if (desStrategy != null){IStrategy strategy = SensitiveStrategyBuiltInUtil.getStrategy(desStrategy);if (strategy != null){if (value.getClass().isArray()){return desArray(value,strategy);} else if (value instanceof Collection) {return desCollection(value,strategy);}else {return strategy.des(value);}}}return value;}/*** 处理脱敏信息** @param field 当前字段*/private Object handleSensitive(final Object originalFieldVal, final Field field) {// 原始字段值IStrategy strategy = null;//处理 @SensitiveMapSensitiveMap sensitiveMap = field.getAnnotation(SensitiveMap.class);if (null != sensitiveMap) {String[] entry =sensitiveMap.value();if (entry != null && entry.length>0 && (entry.length & 1) == 0){// 不为null 且长度一致则将用户指定的脱敏规则加入本次脱敏上下文Map<String,String> map = sensitiveContext.getMapDesStrategy();if (map == null){map = new HashMap<>();sensitiveContext.setMapDesStrategy(map);}for (int i = 1; i<entry.length;i+=2){map.put(entry[i-1],entry[i]);}}return originalFieldVal;}// 系统内置自定义注解的处理,获取所有的注解Annotation[] annotations = field.getAnnotations();if (null != annotations && annotations.length > 0) {strategy = SensitiveStrategyBuiltInUtil.getStrategy(annotations);}// 判断是否获取到指定脱敏规则，如有则进行脱敏处理if (null != strategy){Class fieldTypeClass = field.getType();if(fieldTypeClass.isArray()) {// 为数组类型return desArray(originalFieldVal,strategy);}else if (Collection.class.isAssignableFrom(fieldTypeClass)){// 为集合类型return desCollection(originalFieldVal,strategy);} else {// 普通类型return strategy.des(originalFieldVal);}}return originalFieldVal;}/*** 处理数据类型，根据元素依次遍历脱敏* @param value 未被脱敏的原对象* @param strategy 脱敏策略* @return 脱敏后的新对象*/private Object desArray(Object value,IStrategy strategy){Object[] arrays = (Object[]) value;if (null != arrays && arrays.length > 0) {final int arrayLength = arrays.length;Object newArray = new Object[arrayLength];for (int i = 0; i < arrayLength; i++) {Array.set(newArray, i, strategy.des(arrays[i]));}return newArray;}return value;}/*** 处理集合类型，根据元素依次遍历脱敏* @param value 未被脱敏的原对象* @param strategy 脱敏策略* @return 脱敏后的新对象*/private Object desCollection(Object value,IStrategy strategy){final Collection<Object> entryCollection = (Collection<Object>) value;if (null != entryCollection && !entryCollection.isEmpty()) {List<Object> newResultList = new ArrayList<>(entryCollection.size());for (Object entry : entryCollection) {newResultList.add(strategy.des(entry));}return newResultList;}return value;}
}
上下文对象 SensitiveContext，目前里面只有一个Map，用来存储对map类型元素进行脱敏的策略规则
package com.zhibo.log.sensitive.core.context;import java.util.Map;/*** @Author: Zhibo* @Description: 脱敏上下文*/
public class SensitiveContext {private SensitiveContext(){}/*** Map中Key的指定脱敏规则*/private Map<String,String> mapDesStrategy;public Map<String, String> getMapDesStrategy() {return mapDesStrategy;}public void setMapDesStrategy(Map<String, String> mapDesStrategy) {this.mapDesStrategy = mapDesStrategy;}/*** 新建一个对象实例* @return this*/public static SensitiveContext newInstance() {return new SensitiveContext();}
}
脱敏工具类使用入口
支持自定义Bean 脱敏 根据注解规则；
有时候日志打印的直接就是一个Map，没有写注解的地方，这里也支持了；
还有很多场景直接打印 参数 直接就是一个String的文本，所以也支持 直接对文本进行脱敏；
当然还有数组、集合的直接打印需求也都支持了；
package com.zhibo.log.sensitive.core;import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.serializer.ContextValueFilter;
import com.zhibo.log.format.LogSensitiveConstants;
import com.zhibo.log.sensitive.api.IStrategy;
import com.zhibo.log.sensitive.core.context.SensitiveContext;
import com.zhibo.log.sensitive.core.support.filter.SensitiveContextValueFilter;
import com.zhibo.log.sensitive.core.util.strategy.SensitiveStrategyBuiltInUtil;
import org.apache.commons.lang3.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
import java.util.regex.Pattern;/*** @Author: Zhibo* @Description: 脱敏工具类*/
public final class LogDesensitizeUtil {private static final Logger log = LoggerFactory.getLogger(LogDesensitizeUtil.class);private LogDesensitizeUtil(){}/*** 返回脱敏后的对象 json* null 对象，返回字符串 "null"* @param object 对象/map 自定义的对象，依据对象内注解进行脱敏* @param entry 用于构建一个Map 传递格式 "key1","value1","key2","value2".... 如非map对象可以忽略* @return 结果 json,如处理异常则直接返回原对象*/@SuppressWarnings({"unchecked", "rawtypes"})public static String desJson(Object object, String... entry) {try {//String类型则直接调用desStringif (object instanceof String){return desString(object.toString());}Map<String,String> map = null;if (object!= null && entry != null && entry.length>0 && (entry.length & 1) == 0){map = new HashMap<>();for (int i = 1; i<entry.length;i+=2){map.put(entry[i-1],entry[i]);}}return desJson(object,map);} catch (Exception e) {log.warn("对象脱敏失败 desJson异常 异常信息：",e);return JSON.toJSONString(object);}}/*** 返回脱敏后的对象 json* @param object 需要被脱敏的对象 or Map* @param desStrategy Map中指定Key的脱敏策略，此策略只针对String 类型的值进行脱敏，如Map中存储的是对象，请使用注解进行标记* @return 结果 json,如处理异常则直接返回原对象*/public static String desJson(Object object, Map<String, String> desStrategy) {try {if (null == object) {return JSON.toJSONString(object);}final SensitiveContext context = SensitiveContext.newInstance();context.setMapDesStrategy(desStrategy);ContextValueFilter filter = new SensitiveContextValueFilter(context);return JSON.toJSONString(object, filter);} catch (Exception e) {log.warn("对象脱敏失败 desJson异常 异常信息：",e);return JSON.toJSONString(object);}}/*** 通过正则匹配，返回脱敏后的内容，当前支持11位手机号、18位身份证号码、地址信息匹配* 如已知需脱敏的数据类型,请使用{@link LogDesensitizeUtil#desString(String, String)}方法，* @param value 未脱敏的明文* @return 结果 已脱敏的文本*/public static String desString(String value) {try {if (StringUtils.isBlank(value)){return value;}else if (value.length() == 11){// 匹配手机号规则if (Pattern.compile(LogSensitiveConstants.PHONE_REGEX).matcher(value).matches()){return desString(value, LogSensitiveConstants.STRATEGY_PHONE);}}else if (value.length() == 18){// 匹配身份证号码规则if (Pattern.compile(LogSensitiveConstants.ID_NO_REGEX).matcher(value).matches()){return desString(value, LogSensitiveConstants.STRATEGY_ID_NO);}}} catch (Exception e) {log.warn("数据脱敏失败 desString异常 异常信息：",e);}// 未命中任何规则直接返回明文return value;}/*** 依据指定的脱敏策略返回脱敏后的内容* @param value 需要被脱敏的文本* @param type 指定脱敏策略，详见{@link LogSensitiveConstants}，* 如脱敏策略不存在，则不进行脱敏处理* @return 结果 已脱敏的文本*/public static String desString(String value, String type) {try {if (StringUtils.isNotBlank(value)){IStrategy strategy = SensitiveStrategyBuiltInUtil.getStrategy(type);return null == strategy? value : strategy.des(value);}} catch (Exception e) {log.warn("数据脱敏失败 desString异常 异常信息：",e);}return value;}/*** 依据指定的脱敏策略返回脱敏后的内容* @param values 需要被脱敏的文本* @param type 指定脱敏策略，详见{@link LogSensitiveConstants}，* 如脱敏策略不存在，则不进行脱敏处理* @return 结果 已脱敏的文本*/public static String desString(String[] values, String type) {try {IStrategy strategy = SensitiveStrategyBuiltInUtil.getStrategy(type);if (null != values && values.length>0 && null != strategy){StringBuilder sbd = new StringBuilder("[\"");sbd.append(strategy.des(values[0])).append("\"");for (int i = 1;i<values.length;i++){sbd.append(",\"").append(strategy.des(values[i])).append("\"");}sbd.append("]");return sbd.toString();}} catch (Exception e) {log.warn("数据脱敏失败 desString异常 type：{} 异常信息：",type,e);}return JSON.toJSONString(values);}/*** 依据指定的脱敏策略返回脱敏后的内容* @param values 需要被脱敏的文本* @param type 指定脱敏策略，详见{@link LogSensitiveConstants}，* 如脱敏策略不存在，则不进行脱敏处理* @return 结果 已脱敏的文本*/public static String desString(Collection<String> values, String type) {try {IStrategy strategy = SensitiveStrategyBuiltInUtil.getStrategy(type);if (null != values && values.size()>0 && null != strategy){StringBuilder sbd = new StringBuilder("[");for (String entry : values) {sbd.append("\"").append(strategy.des(entry)).append("\",");}sbd.setCharAt(sbd.length()-1,']');return sbd.toString();}} catch (Exception e) {log.warn("数据脱敏失败 desString异常 异常信息：",e);}return JSON.toJSONString(values);}
}
测试Demo
package com.zhibo.demo;import com.zhibo.log.sensitive.annotation.SensitiveMap;
import com.zhibo.log.sensitive.annotation.strategy.SensitiveStrategyIdNo;
import com.zhibo.log.sensitive.annotation.strategy.SensitiveStrategyPhone;
import com.zhibo.log.sensitive.core.LogDesensitizeUtil;
import com.zhibo.log.format.LogSensitiveConstants;
import lombok.Data;
import lombok.extern.slf4j.Slf4j;import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;/*** @author zhibo*/
@Slf4j
@Data
public class DesDemoBean{/** 对身份证进行脱敏*/@SensitiveStrategyIdNoprivate String idNo = "421083202411111111";/** 对手机号集合进行脱敏 */@SensitiveStrategyPhoneprivate List<String> mobileList = Arrays.asList("18611111111","18622222222","18633333333");private String adderss = "广东省深圳市南山区牛马大厦";/*** 对Map进行脱敏* 传递格式 "key1","value1","key2","value2".... key 为Map中需要脱敏的key，value为脱敏规则*/@SensitiveMap({"idNo", LogSensitiveConstants.STRATEGY_ID_NO,"phone", LogSensitiveConstants.STRATEGY_PHONE, "phoneArr", LogSensitiveConstants.STRATEGY_PHONE})private Map<String,Object> map = new HashMap<>();{map.put("name","吕志博");map.put("phone","18123456789");map.put("phoneArr",new String[]{"18123456780","18123456781","18123456782"});map.put("idNo","421083202411111111");}public static void main(String[] args){System.out.println("------------- 通过注解为对象脱敏Begin ------------------");DesDemoBean user = new DesDemoBean();// LogDesensitizeUtil.desJson 为自定义Bean的专用脱敏方法System.out.println(LogDesensitizeUtil.desJson(user));System.out.println("------------- 通过注解为对象脱敏End ------------------");System.out.println("------------- 通过工具类为Map脱敏Begin ------------------");Map<String,Object> desDemoMap = new HashMap();desDemoMap.put("name","吕志博");desDemoMap.put("phone","18888888888");desDemoMap.put("idNo","421083202411111111");desDemoMap.put("DesDemoBean",user);// 写法一 直接传入需要脱敏的key和脱敏规则， 传递格式 map, "key1","value1","key2","value2".... key 为Map中需要脱敏的key，value为脱敏规则System.out.println("写法一：" + LogDesensitizeUtil.desJson(desDemoMap,"idNo", LogSensitiveConstants.STRATEGY_ID_NO,"phone", LogSensitiveConstants.STRATEGY_PHONE));// 写法二 自行构建脱敏规则，然后以map形式传入Map<String,String> strategyMap = new HashMap<>();strategyMap.put("idNo", LogSensitiveConstants.STRATEGY_ID_NO);strategyMap.put("phone", LogSensitiveConstants.STRATEGY_PHONE);System.out.println("写法二：" + LogDesensitizeUtil.desJson(desDemoMap,strategyMap));System.out.println("------------- 通过工具类为Map脱敏End ------------------");/*** 指定脱敏策略进行脱敏 支持String、String数组、Collection<String>* @param1 需要被脱敏的文本* @param2 指定脱敏策略，详见{@link LogSensitiveConstants}，脱敏策略不存在，则不进行脱敏处理*/System.out.println("对手机号进行脱敏："+LogDesensitizeUtil.desString("18888888888",LogSensitiveConstants.STRATEGY_PHONE));System.out.println("对手机号集合进行脱敏：" + LogDesensitizeUtil.desString(Arrays.asList("18888888888","18888888889"),LogSensitiveConstants.STRATEGY_PHONE));System.out.println("对手机号集合进行脱敏：" + LogDesensitizeUtil.desString(new String[]{"18888888888","18888888889"},LogSensitiveConstants.STRATEGY_PHONE));/*** 通过正则匹配模式对身份证、手机号、地址进行脱敏*/System.out.println("对身份证号码进行正则匹配脱敏：" + LogDesensitizeUtil.desString("42108320241111111X"));System.out.println("对手机号码进行正则匹配脱敏：" + LogDesensitizeUtil.desString("18888888888"));}
}
内容输出如下
------------- 通过注解为对象脱敏Begin ------------------
{"adderss":"广东省深圳市南山区牛马大厦","idNo":"421083********1111","map":{"phoneArr":["181****6780[0907ddf0d173216301559631350fa9ba]","181****6781[54ea4b6a5c8e10eac4ef873e4ce14f25]","181****6782[3f52919f044875b182bc5e6b6ba37271]"],"phone":"181****6789[093b20e8d401ee8309534de0d92eb497]","name":"吕志博","idNo":"421083********1111"},"mobileList":["186****1111[d270298c22d999895d58a1e9fd9d0751]","186****2222[2b035cebca7b1552d48db40778c15863]","186****3333[e2171fb6ec098bd41065098dc7cd6d5b]"]}
------------- 通过注解为对象脱敏End ------------------
------------- 通过工具类为Map脱敏Begin ------------------
写法一：{"phone":"188****8888[cbd41c6103064d3f0af848208c20ece2]","name":"吕志博","idNo":"421083********1111","DesDemoBean":{"adderss":"广东省深圳市南山区牛马大厦","idNo":"421083********1111","map":{"phoneArr":["181****6780[0907ddf0d173216301559631350fa9ba]","181****6781[54ea4b6a5c8e10eac4ef873e4ce14f25]","181****6782[3f52919f044875b182bc5e6b6ba37271]"],"phone":"181****6789[093b20e8d401ee8309534de0d92eb497]","name":"吕志博","idNo":"421083********1111"},"mobileList":["186****1111[d270298c22d999895d58a1e9fd9d0751]","186****2222[2b035cebca7b1552d48db40778c15863]","186****3333[e2171fb6ec098bd41065098dc7cd6d5b]"]}}
写法二：{"phone":"188****8888[cbd41c6103064d3f0af848208c20ece2]","name":"吕志博","idNo":"421083********1111","DesDemoBean":{"adderss":"广东省深圳市南山区牛马大厦","idNo":"421083********1111","map":{"phoneArr":["181****6780[0907ddf0d173216301559631350fa9ba]","181****6781[54ea4b6a5c8e10eac4ef873e4ce14f25]","181****6782[3f52919f044875b182bc5e6b6ba37271]"],"phone":"181****6789[093b20e8d401ee8309534de0d92eb497]","name":"吕志博","idNo":"421083********1111"},"mobileList":["186****1111[d270298c22d999895d58a1e9fd9d0751]","186****2222[2b035cebca7b1552d48db40778c15863]","186****3333[e2171fb6ec098bd41065098dc7cd6d5b]"]}}
------------- 通过工具类为Map脱敏End ------------------
对手机号进行脱敏：188****8888[cbd41c6103064d3f0af848208c20ece2]
对手机号集合进行脱敏：["188****8888[cbd41c6103064d3f0af848208c20ece2]","188****8889[3639d5bc5f940edd8800fb7e7f5a15ba]"]
对手机号集合进行脱敏：["188****8888[cbd41c6103064d3f0af848208c20ece2]","188****8889[3639d5bc5f940edd8800fb7e7f5a15ba]"]
对身份证号码进行正则匹配脱敏：421083********111X
对手机号码进行正则匹配脱敏：188****8888[cbd41c6103064d3f0af848208c20ece2]
